At least from my recollection, commercial rail has an incredibly high amount of regulation and union involvement. Which means a minimum 2 person onboard requirement. I can’t imagine it being an easy market to break into, despite all the technology being there. Wabtec, the locomotive giant, today has TripOptimizer which is smart cruise control, which might be all you really need for most situations. Barring release of sand and operation in the rail yard.

Commercial trucking is largely the same (safety rules and unions)

The problem is not union or regulation. The problem is that American rail infrastructure is a complete disaster and then train companies use that infrastructure to push incredibly long trains on them driven by crews that have horrible working experience (crew switches in the middle of nowhere in the middle of the night and so on).

Having 1 driver is quite common in most of the world but most places don't run on that kind of infrastructure with that kind of loads. 2 crew has prevented massive desasters in US rail history.

If anybody was serious about logic, upgrading the rail infrastructure, totally new signaling system, electrification, shorter trains would allow for a huge amount of self driving.

But don't expect that rail companies will do any of that. They have very little punishment for there trains falling over, or delay caused by Amtrek and they can do with their crews as they like.

Could it be, that just for the potential of a malfunction or accident, it’s beneficial to have a human on board to handle such situations?

And if that’s the case, the human might as well just drive the train while he’s waiting.

The same is true for tractor trailer trucks.

Of course we can automate trains. I'm working in a research project doing that right now on a rail track between the Netherlands and Germany. It's quite a bit of work but I don't see any fundamental issues.

Straightforward enough:

https://www.youtube.com/watch?v=hf5iBbmas2w

... until it isn't:

https://www.youtube.com/watch?v=Ec8Yx-Mp-5s

Linked more for your interest than to make any point - the derailmant issues in the second link weren't directly linked to automation, but we do have big train sets here in W.Australia.