I still don't understand "in simulation", though, because of this line:

> But the company highlighted eight incidents that it considered “most severe or potentially severe.” Three of these crashes occurred in real life and five only in simulation. Airbags were deployed in all eight incidents.

There were 8 real accidents on the road. In all 8 of those real accidents the airbags were deployed. Safety drivers were in some of those cars and took control of the vehicle at some point. When the safety drivers in the car took control; they performed evasive maneuvers that reduced the potential outcome from "most severe or potentially severe" to something less severe.

When the engineers reviewed all of these 8 crashes they played them back watching what the humans did, then they put all of the constraints into their simulation and let the AI take over. When they say "Three of these crashes occurred in real life and five only in simulation" that means that 3 severe crashes happened in real life and the AI would have cause 5 more "severe" crashes had the humans not taken control.

From the second paper[0]:

'In order to provide more information about event severity within the S1 designation, S1 severity events have been separated into two columns in Table 1 based on whether each event is of sufficient severity to result in actual or simulated airbag deployment for any involved vehicle. Of the eight airbag-deployment-level S1 events, five are simulated events with expected airbag deployment, two were actual events involving deployment of only another vehicle’s frontal airbags, and one actual event involved deployment of another vehicle’s frontal airbags and the Waymo vehicle’s side airbags. There were no actual or predicted S2 or S3 events'

[0] https://storage.googleapis.com/sdc-prod/v1/safety-report/Way...

That is quite confusing, yes. Did they mean 5 simulated air bags?

What doesn't make sense? Surely it shouldn't be too hard to even say "air bags deploy when there's x amount of force, so they would have deployed in the simulation" let alone have an actual physics simulation going on.

In all of the incidents, air bags deployed. 5 of the 8 incidents were in simulation. The simulations likely have extremely realistic models for safety critical features such as airbag deployment (which relies on specific sensors in the vehicle).

Or they just looked at what portion of the car got in the simulated collision and and were like "yup, that's where the bag sensor is".

Simulated collision is an important stat. It shows how much improvement the AI needs to obtain to be on-par with a human driver.

Almost, it should how much improvement the AI needs to avoid collisions a human driver did avoid. The AI might avoid collisions that the human didn't.

In other words, you need 0 simulated collisions to be strictly better than a human driver (in the same setting). Not ~equal with sometimes better and sometimes worse.