You are not wrong, but besides the upper-middle-class hobby people, there is also a 130 years old culture that exists parallel to it. I've met a few native Esperanto speakers, and for them Esperanto is their identity. Traditional Esperanto clubs exists in countries like Iran, Japan, China, Burundi, Nigeria and many more. So Esperanto is both, a nerdy hobby and an old culture.