I wonder if it's about the expenditure. If you spend a bunch of money on something, it's probably natural to get more serious/nervous about it. (Though, for all I know, maybe rock collecting is expensive too.)
Or maybe it's about how serious it is to a layman. Rocks are serious! Geologists write papers and hold conferences! Horses, stamps, model trains, people are passionate about them, but it's maybe a little funnier to the layman.
Even just getting riding lessons is $30/hour in a cheap area in a low-cost facility. You can get a good horse for free, but board is $250/month, $500 for everything you need if you buy it used, plus vet and farrier bills. Definitely a higher initial cost and I sure don't buy $250 of rocks a month.
But I don't think rocks are somehow more serious than horses. There's a lot to know about each and there's a ton of money to be spent on either. I honestly think it's just a culture thing. Horses come with status traditionally and rock collectors tend to just like rocks and then drinking a beer and talking about rocks. There's lots of down-to-earth horse owners and I'm sure there's some pompous asses in the rock collecting world. They just don't seem to be the majority.
Seriously, though, you just described my experience of the coin collecting community pretty well, too. You've got the people who have full collections of mint state Buffalo nickels, and you've got the people who collect coins with animals on them. A full boat of high grade Buffalo nickels can run you well over $50k, while you might not even have to put $50 into your animal collection to have a hundred nice pieces.
About the only thing the coin community goes hard on generally speaking is cleaning or artificially toning coins (using chemicals to apply fake patinas or color). Those are no-nos, and you won't get any respect by doing them. The exception is ancient coins, which, since they're hundreds or thousands of years old, and frequently come out of the ground, usually need some cleaning to be presentable.
Having spent so much time in the horse world, the "You like rocks? Me too! Look at this one!" Attitude is so refreshing.