It's too easy for rich successful people who have made it to declare that you should follow your passion. We wouldn't hear a quote from the dozens of lesser-known physicists who struggled and wouldn't recommend the field, we just hear the quote from Feynman who was the lucky one.

For instance, I love painting and woodworking, but I suffer no illusions that I would do "well" in those fields, as even far more talented and experienced individuals that I know in those fields are struggling mightily and envy my hobbyist status.