I think I get what you mean: the percentage of people who access the web using a screenreader is not huge. That doesn't mean we should discount it, it's just common courtesy and it's a legal requirement in many circumstances.

But if you take one step back, the demographic is not very small at all! It includes everyone who is a bit older and maybe can't see as well as when they were 23, so they like to increase the contrast a bit. I myself am not very far above 23 and I read hackernews zoomed in to 130%, it's just more readable. Many people have an easier time using a computer by for example making buttons larger so that a mouse can click them faster. There are more examples.

And last but not least: Writing functional UI tests for applications is enabled, in a good chunk by accessibility capabilites built into the gui toolkit. My job would be much harder if that wasn't baked in to most UIs.

So, while I agree with you: let people have fun playing with technology and create new things without having to implement everything from the beginning, I don't think it is fair to just dismiss accessibility concerns like that.