The majority does care though. Every day you'll hear someone complaining about how frustrating it is to figure out how to do something on their computer that should be simpler, like finding a certain program or file. They're complaining about UI. They don't go on hacker news and rant about it, but that doesn't mean they don't care about good UI design.

Most Linux desktops are not visually consistent. Their UI isn't designed intuitively. You put that in front of the average person, and they will have trouble using it.