My argument was a mild push-back, pointing out that it's more difficult in some respects (e.g., you might not have access to some of your preferred professors), and that it _does_ actually happen occasionally.
Do you have any insight into why more people don't take advantage of that opportunity? Anecdotally, before I attended university I didn't value a college education beyond the slip of paper at the end, and even if I had wanted just the education without the degree I wouldn't have known that I could just sit in on my classes.
Because they correctly value the learning to be worth ~$X,000, but the accreditation to be worth ~$YZ,000.
When you put things that way, the 20-40 hours/week of time that goes into getting an education becomes a lot less of a good investment, if the only payoff is that you get to learn.
Yes, this sort of thing is discouraged - but my point is that anyone who wanted to learn what is being taught in CALC 405, or what-have-you can currently do so, under the table, for essentially free.