While I totally agree about not collecting as an investment, "because you really enjoy the objects as a person who likes this stuff" is probably not actually a terrible metric for future desirability of an item in a niche area of collecting. Other fans/collectors likely feel the same. Of course, it doesn't tell you if that niche will die out and no one will want anything, although I suspect rare books are less at risk of that.

As a collector of memorabilia for a band that's been around a while: The items I was drawn to, either because I just loved the look or because they were from a show/tour/album I particularly found special, have often turned out to be the ones that have appreciated much more in value than the average thing of that level of rarity for the band.