My read is he's saying for the most part mathematics arises in empiricism, in real world problems. Once the problem is translated into mathematics useful tools develop, but then over time branches will spawn that become increasingly abstract/aesthetic and are generally a waste of time.

I also liked the beginning bit where he says you generally can't understand something until you have "assimilated it in an instinctive and empirical way". You need that muscle memory, even for intellectual subjects.