I obviously don't know if this applies for the Fil-C bugs but what we saw in Rust is that there's plenty of "technically that's not OK" which is very distant from "this will actually cause real world problems" https://forge-blog.github.io/posts/rust-std-memory-safety/

For example it's definitely not OK that you can impl Pattern in a way that causes non-UTF8 data in a string with safe code. But you probably didn't let random attackers impl Pattern, on a nightly Rust where that's even possible, and so that's not going to cause problems in our real world - even though it's wrong and must be fixed.

Most of the bugs were things that fall into the “won’t cause real world problems” category.

But they were all things that unambiguously violated the Fil-C promise

I am speculating on a limb here: maybe the Mythos zero days developed into exploits had a few of the "usually won't cause real world problems" happenstances.

But a model with all the world's knowledge would know how to weave them together and successfully exploit.

I like to think that tomorrow's models are going to be more capable, what we are experiencing now is a fire drill to prepare for that.

> One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary.

Building software boils down to defining things correctly. So, it sounds like you did most of the work here. Imagine that you took your definition and moved it into a formal specification (programming language) and then ran that to verify the security properties of your system. What's different?

Or, imagine that you took your original code as your specification language and then iterated on your understanding of it until it was correct. What's different?

Attention highlights and resolves bugs, so that's the baseline. In order for AI to get value, it has to be some kind of multiplier on your attention, but the reality is generally the opposite.