The trade off is that the pedestrian has pretty much no right of way anywhere but a crossing, and cars will drive at you (or at least not stop for you) if you try to cross somewhere that is not a crossing. Though "Jaywalking" is not a thing and you can actually cross where ever you like.
That's not true, or at least it's bad/illegal driving if they do so, a pedestrian who is 'established in the road' as the right of way anywhere.
As you said:
> Though "Jaywalking" is not a thing and you can actually cross where ever you like.
Otherwise it would be a contradiction wouldn't it? If the pedestrians allowed to be there, the motorist obviously isn't allowed to run them over, ... I suppose you could say the pedestrian can continue crossing but only after first giving way to the motorist? There'd be more time with pedestrians in the road though.
Not that I recommend using that fact to cross when you don't have time, because you will anger motorists. Or they could not see you/be paying attention. They'd be wrong, but it's just not worth it, obviously.
It's illegal in most if not all of USA too, but no one cares in practice. Legally, even when a car driver and a pedestrian both have access to a lane separately, if both are present, then a car driver must give a full lane-width of space to a pedestrian crossing or at the corner.
Also, even when a pedestrian is committing the auto-industry-invented crime of "jaywalking", the pedestrian still has the right of way in traffic, unless it is physically impossible for the car driver to avoid the collision. Car drivers are not judge/jury/executioner.
(Nit: "Cars" don't "drive" (yet, in most places). "Car drivers" drive cars.)
"Cars driving at you" is probably a dialectical thing.