You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.

This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.

This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?

True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.

There is even a contest for it: https://www.underhanded-c.org/

Yes, "this looks like an underhanded c challenge" was my first thought when reading this. See one mistake by one project, inject the matching intentional mistake into another project (that also conveniently fixes some known bug), and boom - plausibly deniable backdoor.

If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").

> humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances,

Not everyone is like your sister and mother.

actually this looks very underhanded indeed, what are the odds they pick the same sentinel character?!

Like 1 in 5? The set of go-to characters for that kind of thing is pretty small.

Why restrict yourself to a single sentinel character?

Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?

Be careful, saying this gets you downvoted like https://news.ycombinator.com/item?id=50364646

Never attribute to malice that which yada yada yada

And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印材 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.

No the biggest lesson of all this AI stuff to me is that we just suck at programming

The whole idea of neural nets is that we aren’t smart enough to design the function as programmers so we need to learn it

No, the idea of neural nets is that writing smart functions doesn't scale, not that it's not possible. Take eg. Chess: the possibility space is simply too large to write a smart function for every possibility.