Then drive slower. It's a speed limit not a speed minimum.
When I was training for my drivers license (in Denmark) my instructor told me to never drive so fast that I wouldn't be able to break to a complete stop within the current visible distance.
On highways, enough clearance from the sides of the road and wide enough detection beam cone may be able to help. Yes, I imagine in such a case high-speed motorways would warrant security measures similar to those around high-speed train tracks.
On denser city streets where restricting pedestrian access is not feasible nor desirable[0], appropriate (yes, probably pretty low) speed limits could reduce braking distance and hence the chance of a tragic accident to below what currently happens with human drivers.
[0] This concerns existing streets. For new areas, infrastructure that separates car and pedestrian traffic with tunnels/bridges and accessible over-/underpasses could address that naturally.
> An autopilot should not be entrusted to decide further, beyond anomaly detection sufficiently far ahead to allow for braking.
This isn't possible. Not just technical but theoretical. Situations in streets can change way to fast. Both through external factors and other cars. Even on a straight highway with fences on the side to prevent anything from going on it this would still but work. Once it happens an intermediate decision is needed which can not be reached on time by delegation to the driver (as the driver want driving you can't except a good reaction speed even if that person had the eyes on the road).