I too, started late in the game. With a brief introduction to Basic in school, and some programming classes in undergrad, I graduated with a Bachelors in Production Engineering, and it was after that I realized that computers (and programming) is where I felt just right. I worked on my first major (Java) project in grad school when I was 21. Haven't looked back since and over the years have learned several languages.

I will admit though, that a)starting late b)not having a formal education in computer sciences is a little frustrating at times. I spend my evenings trying to play catch up with those who had a head start (reading up on compilers, language design, even SICP). Don't get me wrong, I am still as fascinated with computers as I was 8 years ago (I turn 30 today, the 1st actually), but every time I read a brilliant comment on a language or a framework on YC News, I know I have a long way to go before I will have that level of competence and understanding.

To answer your question, I don't think its never too late to pursue your passion. In the long run, when you look back, you would rather see having spent time doing things you loved to do, rather than spending a majority of your life as a zombie, in pursuit of a paycheck than anything else.

From another late starter, I can only wish you the very best.