If you’ve just downed your fifth cup of coffee and you seem to be hearing things that aren’t actually there, well, you might want to blame the caffeine in your cup of Joe…

Australian researchers found that drinking five regular cups of caffeinated coffee may raise the risk of auditory hallucinations, according to a small study published in the journal Personality and Individual Differences.

LaTrobe University researchers asked 92 people with varying levels of caffeine intake and stress listen to three minutes of white noise via headphones. The study participants were told that Bing Crosby’s “White Christmas” would be playing in the headphones, and when they heard it, to hit a buzzer.

While the song never actually played, the people who were hopped up on five or more cups of coffeebuzzed three times on average, while the low-caffeinated people buzzed just once on average, theHerald Sun in Australia reported.

Though this particular study size was small, it’s worth noting that the findings match up with that of a study published in 2009 in the same journal. That study showed that highly caffeinated people — those who had consumed the equivalent of three brewed cups of coffee — were three times more likely to hear or see things that aren’t there, LiveScience reported.

The theory is that caffeine may worsen the effects of stress on the body. The body naturally releases a hormone called cortisol when it is under stress, but caffeine seems to increase the amount of cortisol that is released.