pick the team to join solely based on the manager

My experience too. Of course there are other factors, but this is (arguably) the most important factor of all; the rest are just constraints, and this is like the objective function.