If you're a baseball player, you don't want to work for a football team. You'll be undervalued and hate your job.

Less metaphorically, I've always looked for jobs where my skills are aligned with the "company mission" - first at a bunch of startups, and now as an academic researcher where I get to define that mission.