When you get older, read more, travel more, experience more, you'll understand the value of a well-rounded education.
Or maybe not. Perhaps confining yourself to a one-track life and career path will prevent you from seeing, or being considered for, other opportunities in life. You'll never know.
I'm using "young life" in the sense of "up to that point in my life". I'm now in my mid-30s and on my 3rd career (first in sales, then as an overseas ESL teacher, and now as a software developer). I've travelled to 20+ countries, lived in China and Europe, and had many wonderful experiences along the way. I'm grateful for the many careers I've had, which allow me to state with confidence that my current career is a great fit for me. But none of my experiences required that I attend the college I did and go into deep debt to do so. While (as you said) I'll never know for sure, my guess is that debt probably cut off just as many opportunities as it gave me.
Knowing what I know now, I would have jumped at the chance to study for free in a place like Germany (as per danjoc's comment). That sounds like the perfect way to minimize the costs of what college purports to offer- a chance to broaden one's horizons and investigate possible directions one wants to take with their life.