Maybe it depends on how the milk was processed or not rather than the milk itself? I also have health issues with milk, yet find yoghurt and milk based products just fine.

There are studies that found a link between gut bacteria and mental illnesses. Some of the bacteria in our guts also thrives in milk based products like yoghurt.