Even taking that into account, I usually take the train when I'm going to SF on business. Even though the schedules are such that one time I had to freeze my butt of on a platform for 50 mins because there were no trains at all during this time to my direction. But at least the option is there, and at least in SF you can get somewhere with public transit. Somewhere in Santa Clara county, usually you don't have any imaginable option to use public transit unless you are ready to spend 2.5 hours on a trip that is 20-minute car ride otherwise.
In addition, there are only a total of eleven of those baby bullet trains, per day. At many hours of the day, you can't get one at all.
On average, given the average wait time til the next train and other factors, I can drive to SF about 2.5 times faster, round-trip, than I can reach various common destinations using CalTrain plus whatever else is available.
It's true that if you are commuting in rush hour only, versus heavy traffic, and your destination happens to be very close to a CalTrain station, and you live close to the CalTrain station to begin with, the train will win. But not under most other conditions.