Cars waste exactly as much specific energy coming to a stop from a given speed as a train (or anything else) does. City speeds are limited precisely because otherwise the stopping distance would be too great. If you're driving 25mph in a city and someones jumps out in front of you and you hit them, you're not going to get blamed. If you're driving 65mph you will.

It only "makes sense" for trains to have unimpeded right of way if you're prioritizing corporate expedience over human life. Within my point there are still two main ways trains can move at high speeds - bridge crossings and crossing gates with positive control. If railroads want to operate at high speeds, then they should have to fix their infrastructure rather than being allowed to continue to dumping their externalities onto the public.