I think your feeling is probably the most common one, which is why short science fiction readers are a vanishingly small percentage of the population. One of the reasons novels are much more popular than short fiction (orders of magnitude more popular) is because once you find a world you enjoy you can sit in it for a while. With short fiction as soon as you build up the world in your head it's done and you have to move on to the next one.

I like jumping from world to world more than the average person -- I'm happy getting that new, novel idea and then jumping to the next thing. I understand I'm atypical, but I think there's probably a higher percentage of people like me on HN than there are in the overall population.