It's probably due to being able to digest lactose as a benefit it's not a problem so that throws off our perception of it.

If 5% of the world developed the ability to see UV light would be call the other 95% UV-intolerant.

For the record I am not lactose intolerant my northern European ancestors developed that ability (plus blue eyes, blond hair & red hair).