The most likely culprit: lactose.
The British Medical Journal published a large study that tracked the dietary habits of over 100,000 people in Sweden. The study followed the participants for 20 years taking into account differences in lifestyle and demographics. The researchers at Uppsala University found that women who drank three or more glasses of milk per day were twice as likely to die earlier than those who drank less than one glass of milk per day; men had a slightly higher risk of dying early as well. The study also found that the more milk the women in the study drank, the more likely they were to experience bone fractures, especially hip fractures.