Even just getting riding lessons is $30/hour in a cheap area in a low-cost facility. You can get a good horse for free, but board is $250/month, $500 for everything you need if you buy it used, plus vet and farrier bills. Definitely a higher initial cost and I sure don't buy $250 of rocks a month.
But I don't think rocks are somehow more serious than horses. There's a lot to know about each and there's a ton of money to be spent on either. I honestly think it's just a culture thing. Horses come with status traditionally and rock collectors tend to just like rocks and then drinking a beer and talking about rocks. There's lots of down-to-earth horse owners and I'm sure there's some pompous asses in the rock collecting world. They just don't seem to be the majority.
I wonder if it's about the expenditure. If you spend a bunch of money on something, it's probably natural to get more serious/nervous about it. (Though, for all I know, maybe rock collecting is expensive too.)
Or maybe it's about how serious it is to a layman. Rocks are serious! Geologists write papers and hold conferences! Horses, stamps, model trains, people are passionate about them, but it's maybe a little funnier to the layman.