With regard to your last paragraph. I believe it is the amount of money that is the corrupting influence. It's not college athletics per se but rather the massive amount of money spent on certain sports that generates problems. The extracurricular activities you mentioned aren't bad or detrimental to the institution's purported goal of education because there's no money corrupting those activities.
With that said, the current system sucks. Many athletes have lost their scholarships due to injury and were then on the hook for the costs of school, requiring some to transfer or drop out. There are some sports and programs that expect most of your time to be dedicated to the sport rather than your academics. The culture around some programs is toxic.
But none of it is inherent to the idea of "college sports". Would you argue that other extracurricular activities like clubs, student government, etc., are detrimental to education? Or would you say that they provide education in other areas of life compared to pure academia?