I think the vehicle should alert the driver, and being controlled slowdown until it is overridden. Sorry, driver, you are inconvenienced. Maybe you are in the carpool / AI-only lane, so it's not a big deal. Yes, there probably should be an AI-only lane for now.

This could exist now, but we have set sights on Level 5 at the expense of pragmatism.