1. Just plain old not knowing better.
2. Being in NYC, my social circle had a lot of Wall Street finance types. In fact, this is still true - I know more finance people amongst my friends than fellow SWEs.
3. Hacker tech culture wasn't as widespread or widely known back then, especially in NYC. Or maybe it was just me. Going back to #1, I thought things couldn't get any better than dressing up in business casual monkey suits every morning and being ordered to dance like a monkey by a hotshot trader in hopes of picking up after his glorious scraps.
4. I thought the money wasn't half bad, making like $80-90k back then. I thought no one could pay better than the Masters of the Universe on Wall Street.
I don't know what a company like Google was paying their engineers back then, but I think it's a somewhat recent phenomenon where tech SWE compensation grossly and utterly outpaced tech-in-finance SWE compensation. Again, I'm not referring to ultra-elite finance firms like Jane Street or Citadel where I understand SWEs do get paid on par (or better) than FAANG.
I distinctly recall even ignoring a Google recruiter's solicitations back then (2007? 2008?) because I naively thought working at my investment bank was utterly superior to anything Google could offer.
Now I know better, and knowing is half the battle.
In the days before the internet, technology was niche (as was the knowledge to develop and operate it) and super expensive, so only mega corps had decent tech to work with.
This reputation persisted for sometime into the new millennium until we started to see more of these scruffy younguns starting to make noise in the business, and techno, spheres.