It's prodding into personal details that many draw the line with. You might not have a problem with it, but I've personally heard from a few different people how they get annoyed with those sorts of questions being leveled at them all the time, simply because they didn't look white. Aziz Ansari even joked about this a few times during Parks and Recreation, it's such a common phenomenon. Imagine seeing someone who might have some sort of disability then asking them if they were insert disability you are unqualified to diagnose but assume they have. That would be considered pretty rude.