Done properly, moving away from #fff/#000 can be done in such a way that it looks excellent on better monitors and is usable, at very least not entirely unpleasant to read, on less capable ones.
> I will have a harder time reading your site
It not looking as nice for you is fine, as long as it is readable. If we held everything back to making sure it worked on all screens there would be a lot more plain-text-only sites/apps out there. A perfectly good option IMO, but try getting the general screen using public to agree with that!
Even with a bad screen there are usually adjustments you can make to help.
There are of course many pages out there that are too low contrast, or otherwise unpleasant, unless you are using a particularly good monitor setup a particular way for which…
> or just give up and go somewhere else.
… is a perfectly valid option. You don't owe them your attention, though likewise they generally don't owe you anything either.
I guess the real problem is that there's no real standard for what #fff means – it has to be interpreted through a colour space, which in turn does not talk about brightness, I think.