People don't always press the buttons, and not all intersections even have buttons. Even with a button press, there's no way to tell the difference between a jogger and a group of elderly people using walkers, so you have to assume the longest possible delay.
That said, there is a thing called a pedestrian scramble intersection [1] with a part of the cycle where all the traffic lights go red and all the walk signals turn on, allowing pedestrians to cross even diagonally. This is relatively rare, only being used at some of the busiest pedestrian intersections in some cities.
Plus a traffic light's sensors wouldn't even moving around in the environment, so it's got to be that much easier. And they can be positioned all over an area, on both sides of obstacles. And they can be higher up for a better view.
The downside is that every intersection is different and drawing conclusions from all that information will be harder. And the stakes are higher with a system that directs everybody else.
I’ve seen the type of light you describe (all pedestrians walk in all directions at once while all cars have a red) in the US but they’re exceedingly rare. In cities there’s always pedestrians and cars waiting in every direction so it would be too slow to be practical.
But why on earth would you turn the tragic light green while the walk signal is still on? Why not one at a time?