But drivers have a duty of care, and if a pedestrian is crossing a road then obviously they must do what is safe, i.e. slow down and let the pedestrian cross if (s)he is in the way.
e.g., where I live, the traffic laws are defined in terms of who has the responsibility to yield, and undergirded by a catch-all requirement that everybody who is able to act to avoid a collision must do so.
For instance in Germany they drill into you in driver's ed that when you hit someone with your car, it's pretty much always your fault. I'd say we don't have a jaywalking epidemic.
The reason some US states choose to specify who should yield instead of who gets to go first is because framing things in those terms hopefully alters the ways that drivers think about things in a way that will improve safety overall.
TBH, I suspect that it's an empty gesture that is being used as an alternative to implementing more rigorous driver education and licensure requirements; the US tends to view driving as more of a right than a privilege.
It's equally reasonable for pedestrians NOT to have priority to start crossing the road unless there is a clear, marked pedestrian crossing.
Priority means that cars must give way, which is the case at pedestrian crossings: Cars must give way BEFORE a pedestrian has started to cross in order to let him cross.
But in any case once a pedestrian is on the road cars have a duty of care not to hit him.
One example would be if you were leading a walking group of 20 people and wanted to cross the road and it was safe to do so at the beginning but because the group is large it takes a long time. Any approaching cars would not have any right of way until all the group has passed.