I think a part of the problem is that most(?) rail lines are owned by a patchwork of companies. So while you can take an Amtrak from San Jose to LA, it will travel over BNSF and Union-Pacific lines (I think). It may not get right-of-way everywhere, resulting in delays.

We need the Federal Government to own the rail lines, just like it owns the highways.