Pure speculation but what if it has to do with dopamine sources? Solving problems and the certainty of math is addictive to some of us and dealing with people may not be. If you ride up the ladder of solving problems until you reach the ceiling of having to deal with people (especially academia politics), a farm where there is certainty and sunshine and where you can still solve problems as you like without having to depend on people, may be attractive.