He's not comparing math to other academic fields, he's comparing it to the most realistic alternative for most people, which is working a garbage McJob in which math isn't relevant to their lives at all. There's a lot of privilege involved in burying your head in books, no matter what the field, barring a few obsessive prodigies who live in public libraries and have extremely supportive parents.