>95%+ of North American intercity trains run on freight tracks, which are not designed to be as "smooth". On top of this, freight having priority means passenger schedules get messed up all the time.

Freight doesn't mean slow.

https://en.wikipedia.org/wiki/Rail_speed_limits_in_the_Unite...