It seems that Comma are actively working on documenting their safety systems (via a new hire to own this task) and creating an audited checklist. And, there is a "new vehicle" checklist enumerated in a Medium post including some safety constraints, although in reading the Panda "safety" tree, I found no actual acknowledgement that these specific constraints were audited for each vehicle addition.
So, it seems that the great strides since 2018 continue - once these promises come true, they will have almost reached what I would consider the baseline level for a safety-related automotive system. Perhaps they will be able to secure an exit this way - I am actually rather surprised they haven't, given the froth level in this space as well as the valuable training data they have collected from their users.
Every self-driving product that deploys to consumers is going to end up in court eventually. It's inevitable.
Imagine trying to defend comma.ai to a lay jury... the prosecution won't even have to try.
Meanwhile, taking these extra rote precautions (e.g. cut down on global state, throw unit tests on everything, don't use go-to's) has no real effect when the elephant in the room is that even correctly-implemented state-of-the-art machine learning models only have classification accuracies in the mid 90%s at best.
That's (relatively) fine if you're deciding what ad to show to a teenager using TikTok. When we are talking about driving a car, that loosely means having an accident every 100 miles. Based on publicly available disengagement data, this corroborates (only Waymo has surpassed the 1000 mile mark, GM might have recently, too).
Meanwhile, verification in this domain only means replaying actual recorded sensor data in a "simulation" using one-off example scenarios, not doing any kind of formal analysis or exhaustive proofs -- which simply don't exist yet for ML. There's no real way currently to take an arbitrary mis-classified example and determine its root cause -- or even correct it within the existing model without reducing accuracy elsewhere!
It's literally impossible to avoid being unsafe, so code style rules and procedures amount to mere hoop-jumping
I disagree, however that code style rules and procedures amount to hoop-jumping, as not every failure mode is the same (especially if you do accept the driver-attention / driver-in-control premise). An unexpected failure at the control layer (especially the control override layer) is a different class of failure to an unexpected direction command from the ML model, for example.
I don't think they have the money to fight multiple Court challenges, even if you're right you still need to fight. I doubt this company will be around in 5 years, and I at least hope anyone who installs this thing in their car and then gets into an accident loses their license.
I'm actually not anti-Comma, just responding to this specific point.
But there's no way I would bring this up in their GitHub, for example, because it isn't the culture of the project - and that's the problem. For example, their "leader" shows up and makes commits to their safety code like "WTF WHY WAS THIS SHIT EVERYWHERE."
To Comma's credit, they have made great strides in safety since the first two years of their product - since 2018, they've introduced safety checks that (while undocumented), seem to be somewhat consistent (or trying to be) across vehicles, are written in MISRA C which is at least something that's actually used in the field, and exist at the low level of the system. Commits are signed now, and their founder seems to make fewer non-documenting troll commits to the product. These are all signs that the company is growing up (and, if we look at the history of the company and their leadership, we certainly see some signs as to why this may be the case...).
But, these are things that should have been in place since day one, and the founders and company were certainly not unaware. It is absolutely fair to judge products and companies for the decisions they make, ESPECIALLY when their product is purportedly "world-changing," and in the case of Comma, safety does not seem ingrained into the decision-making process.
As an aside, this blog post is killing me, between extensive Peter Thiel quotes, random jabs at California's COVID policy, Elon worship, and a lionization of a startup company with an immature founder building safety-related hardware.