Moreso for music than books, but I personally find that most of my favorites tend to have at least as many people that hate it as love it.
Neutral Milk Hotel's "In the Aeroplane Over the Sea", for example, is an extremely polarizing album. Between Mangum's nasally vocals, beginner-to-intermediate technical talent, the inclusion of saws and theramin as instruments; it all adds up to a love/hate affair. I'm not a big fan of any of Mangum's other works, but AOtS has an allure that is just... indescribably gripping.
I don't go around recommending it, but when it comes up in my playlist, I find that I am simply compelled to stop what I'm doing and listen to the entire album, which is thankfully short, as far as albums go.
Thank you. Haven't thought about that album for many years, but now that you brought it back to my attention, I'll listen to it later today - which nicely supports the point you're trying to make.
I remember a counter-example to this (which may be a fluke) a few years ago when Burial got album of the year on metacritic for a spacey, ambient dubstep record. Maybe not super-niche but definitely not mainstream.
I've explained it like this: To get on the list, something has to be considered at-least-good by a lot of people, and this tends to reward (1) herd mentality; (2) lowest-common-denominator. The list selects against anything in any niche, even when it's excellent.
As I look back through music that has meant a lot to me, there is just not much overlap with best-of lists.
Note: I'm talking about critics-vote, pooled, best-of lists. Single-critic best-of lists don't average out niche tastes and, for the right critic-listener match, can be very helpful indeed.