I think it is mostly an issue caused by how managers are evaluated. Do the teams you supervise outperform (in a short-term metric set) the other teams? If so, you are deemed a good manager.

Upper management usually does not seem to care whether the people you manage leave the company eventually or try to change teams to get away from you.