Athletics for most schools are a net profit center, even for the #100 football program. The smaller schools aren't spending nearly so much on athletics as the schools with football programs.
Meanwhile, the coaches are being paid millions of dollars per year. I don't care how much money the athletic program is bringing in; that's not a responsible use of the money. (I get that it's something of a prisoner's dilemma at the individual institution level; I'm more talking about the broader issue.) If you have the money to pay the coaches $2M/year, that means you could, instead, stop paying your adjunct professors and regular staff sub-poverty wages, and/or perform proper maintenance on the buildings.
And do not get me started on the economics of stadium construction.
The money goes to the athletics and the infrastructure. (Not in the sense of "the best infrastructure money can buy", sadly; no, it's more like "raise $300 million to build this new dorm and let the richest donor put their name on it, then spend about 30% of what it needs for maintenance for the next 30 years...and then spend 5x the amount we would've had to if we'd just maintained it in remediation and renovations".)