News for hackers. Stuff that matters.
Re:AI is forcing us to pay off 40 years of technical debt by Balgair on Wednesday November 04, 2026 @03:54PM (#50371375)
Aside: I think I need some help here.[0]

I remember reading Godel Escher Bach back in the day, and there is the parable of the record player between the tortoise and the rabbit. And the tortoise keeps building more and more secure record players that the rabbit keeps finding ways to break.

GEB then goes on to teach you that, no, in fact, you cannot have a perfectly secure system. Not just practically or economically, but even in theory, all systems are fundamentally breakable. GEB, after a lot of chapters (what a doorstop!) ties that parable into Godel's Incompleteness Theorem, and essentially proves that no, you can't make unbreakable things, as a fundamental point of logic.

So, when I look at all this, I see that yes, there is an economics point of view here, that fixing code is just expensive, like making a less and less breakable record player. And that AIs push that breaking and economics point much further along than what we have had.

But then I also see a lot of comments here that seem (to me) to be arguing that there is an unbreakable code out there.

Am I reading this right? Is there such a thing as 'perfect' code? Or is my outdated reading of GEB still valid or not?

[0]Note, I'm not a CS person, more a biophysics person.

Re:AI is forcing us to pay off 40 years of technical debt by zeroonetwothree on Wednesday November 04, 2026 @04:03PM (#50371485)
A specific system could be proven to be secure. There is no general mechanism to prove all software secure however.
Re:AI is forcing us to pay off 40 years of technical debt by tennysont on Wednesday November 04, 2026 @04:11PM (#50371578)
Godel's Incompleteness Theorem basically says "any system complicated enough to give you basic math is either 'incomplete' or broken."

Clearly our math system gives basic math, so we should conclude that it is either 'incomplete' or broken. Before the incompleteness theorem, it was exciting to imagine that it might be complete, but we had no idea. Now it seems likely to be incomplete. The odds of it being broken haven't really changed.

If it is broken, 2+2 still equals 4 (we didn't really learn that fact from abstract math), but we'll need to fix up set theory. In a world where math isn't formally verified (and even lean itself isn't formally verified), the Incompleteness Theorem is permission to be awed by how huge the space of all logical deduction is, not an argument that logic is a fiction.

To come at it another way, do you believe that there is an 64-bit integer which, any time it is inserted into register 8 on any computer, will immediate convert the raw mass of the earth into a black hole? No, because even if you suspect there is a hole in the foundation of math, you still believe in brute-force case-analysis, and you know that every 64-bit integer has been inserted into register 8 on a computer before.

Re:AI is forcing us to pay off 40 years of technical debt by hilariously on Wednesday November 04, 2026 @04:13PM (#50371615)
The important distinction in the book that was made is that it was for a sufficiently advanced reproducer of the original sound, a thing that would make the perfect sound based in the record would cause its own destruction. Something lossy or without that property is actually less susceptible to damage.
Re:AI is forcing us to pay off 40 years of technical debt by wrs on Wednesday November 04, 2026 @04:18PM (#50371680)
[delayed]
Re:AI is forcing us to pay off 40 years of technical debt by goodmythical on Wednesday November 04, 2026 @06:24PM (#50372859)
>But in real life compilers can still detect infinite loops in your code unless you try really hard to fool them.

Having never written compiled code, this intriques me. I often write one offs that I actually want to loop infinitely until I tell them to stop.

How does the compiler know whether it's just really long or infinite? Like, if I write a progressive approximation of pi or prime search algorithm, both are never going to halt because we're certain there are infinite digits of pi and prime numbers. The program could technically halt, but not as a function of the program but rather the hardware.

Or, being silly, if I write a "flip bits in target directory until all bits are flipped" and target /dev/null as a user of the software, it wouldn't halt, so the compiler would detect that?

Is there some --don't-mind-the-recursion flag for developing a program like ycruncher? I get the feeling that there's a list of "this typically causes infinities" that it checks against and if your use isn't on there well then... So it's just the mouse/mouse trap thing but in a compiler?

Or is there some unique CS going on that the compiler employs to determine certain classes of loops?

Re:AI is forcing us to pay off 40 years of technical debt by pillmillipedes on Wednesday November 04, 2026 @08:01PM (#50373656)
to my knowledge, they detect some common intentional and unintentional infinite loop patterns, but not much more:

    for(;;) // intentional

    for(unsigned i=10;i>=0;i--)
    for(int i=0;j<10;i++)
    for(unsigned i=0;i!=100;i+=8)
    // unintentional
in C infinite loops with no side effects are UB, so detecting them is useful for diagnostics.

guaranteed termination also is needed for some optimizations. but since side-effect-less infinite loops are UB in C, a compiler doesn't have to prove that such a loop terminates, it can just assume it does. other languages may differ, but I don't think any of them prove convergence of algorithms.

a don't-mind-the-recursion flag can be `-fno-finite-loops` on clang, or a read or write to a `volatile` variable in standard C.

Re:AI is forcing us to pay off 40 years of technical debt by tarsierscripter on Wednesday November 04, 2026 @08:04PM (#50373688)
How does the compiler know whether it's just really long or infinite?

One straightforward approach is to use Hoare logic [1], specifically the method of loop invariants and loop conditions. If you have a loop where the conditions are expressed only in terms of invariants then the conditions will never change truth values during iteration of the loop, thus you have an infinite loop.

Now, some loops are written deliberately like this, i.e. "while (true)", and so there you need to look at whether the body of the loop contains any side effects (reading input, printing, etc). Then you have what is called a "productive infinite loop", such as an event loop or a REPL. Of course you can't prove this sort of loop will ever halt, since the user may never elect to issue the quit command, but these sorts of loops often tend to spend most of their time with the process sleeping until the operating system wakes them up to handle an event.

[1] https://en.wikipedia.org/wiki/Hoare_logic

Re:AI is forcing us to pay off 40 years of technical debt by zamadatix on Wednesday November 04, 2026 @08:59PM (#50374132)
I think it's worth pointing out the compiler usually won't do much (if any) of this kind of check unless you tell it to.

When you do tell it to, it's usually to warn you that you have accidentally made some really dumb mistake which resulted in a loop which will run forever while doing nothing externally useful to the loop.

Whenever you do something useful with an infinite loop, like come up with additional digits of pi or progressing the search for the next prime number, it results in some form of useful output which touches the environment outside the loop (e.g. print the number to the screen) and so it won't trigger this kind of thing.

Even then, it won't catch every such case. It just so happens that the vast majority of these cases are really easy to show in practice. Something like a loop which does absolutely nothing but loop until it calculates the last digit of pi would be an example of a useless infinite loop that the compiler might not catch, but also an example of one you'd kinda have to go out of your way to make fail in such a way in the first place.

Re:AI is forcing us to pay off 40 years of technical debt by wrs on Thursday November 05, 2026 @12:16AM (#50375493)
Interestingly, in C++, a trivial infinite loop is undefined behavior. So the compiler not only doesn’t have to detect it, it is allowed to assume you didn’t do it. [0] This is inconvenient when you actually want a trivial infinite loop. (C++26 was going to fix this, but it seems we may not ever get a C++26…)

[0] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p28...

Re:AI is forcing us to pay off 40 years of technical debt by stogot on Wednesday November 04, 2026 @09:10PM (#50374220)
That’s the point of the halting problem You can’t know if it will halt absolutely
Re:AI is forcing us to pay off 40 years of technical debt by Tryk on Wednesday November 04, 2026 @06:05PM (#50372701)
This is a misconception of what is being said in GEB.

Gödel's Incompleteness, very roughly, says that not every true statement (in a formal system) can be proven. In particular, it says that for classical logic with some arithmetic the statement holds; how you get to security I do not follow.

Perhaps you are connecting formal systems to computer programs through the Curry-Howard correspondence, but that correspondence is not between classical logic and computer programs but a related but different logic (called intuitionistic logic). In Intuitionistic logic you cannot prove Gödel's Incompleteness.

Re:AI is forcing us to pay off 40 years of technical debt by Balgair on Wednesday November 04, 2026 @11:09PM (#50375051)
Again, not a CS person.

The idea is that since you can make every security program turing complete, I thiiiiink?, then you can have it be a formal system. So, you can then get the code, in very tortured ways, to execute a paradox, thereby breaking it. Likely in enough ways to let you gain access, though not always.

Re:AI is forcing us to pay off 40 years of technical debt by Tryk on Thursday November 05, 2026 @02:11PM (#50383097)
I mean this in the nicest way, but I would never have dreamt to relate Gödel's incompleteness to this topic.

I think perhaps where I would diverge is that we should separate the language from the program. Generally, we say that a programming language is Turing complete.

Gödel's incompleteness is analogous to a paradox, e.g the Liar paradox, that is one useful way to understand the statement. But it seems as if you're saying it's a Contradiction (from where we can derive anything) something like EFQ: https://en.wikipedia.org/wiki/Principle_of_explosion

That does not follow, not all paradoxes are contradictions. For instance, just because Gödel's incompleteness pertains to first-order Logic does not mean that we can derive anything in first-order logic.

Re:AI is forcing us to pay off 40 years of technical debt by y1n0 on Wednesday November 04, 2026 @07:45PM (#50373525)
That’s pretty interesting about the parable you mentioned. Maybe I’ll give GEB a try.
Re:AI is forcing us to pay off 40 years of technical debt by Balgair on Wednesday November 04, 2026 @11:05PM (#50375026)
It is a literal brick, FYI.

But yes, a hard recommendation. Absolutely worth the time. It's also pretty easy to read.

Re:AI is forcing us to pay off 40 years of technical debt by talon8635 on Wednesday November 04, 2026 @11:20PM (#50375135)
Thank you for reminding me of this. What. Wonderful book. I wish my memory were better at recalling it as yours was.
Re:AI is forcing us to pay off 40 years of technical debt by Asooka on Thursday November 05, 2026 @12:50AM (#50375669)
It's Goedel, not Godel. The umlaut is shorthand for "e".
Re:AI is forcing us to pay off 40 years of technical debt by torginus on Thursday November 05, 2026 @05:13AM (#50377400)
Perfect security is actually pretty easy to achieve, for example, if you turn off your computer, it's impossible to hack over the internet. But it's also useless - you want some people to access the machine some of the time.

Another way you have very good (and fake) security, is by reducing the scope of the problem - if you check every request against a very strong crypto signature, it's also practically unbreakable.

But then you have issues like who signs these tokens, who decides which user gets what permissions etc. etc. which mean you have to weaken practical security to make it useful.

Another example is the L4 microkernel, which 'proven' to be bug-free and secure. But it also doesn't do like 99% of the stuff a regular kernel like Linux does, and sort of pushes the issue to the user.

So even defining 'perfect' security in a way that the system remains practically usable is very difficult and potentially impossible.

We come down to to the permissions problem - each person should have as much permissions to do their work as necessary and nothing more - which is essentially a reformulation of Gödel's incompleteness theorem.

Re:AI is forcing us to pay off 40 years of technical debt by abetusk on Thursday November 05, 2026 @10:13AM (#50380110)
Godel's incompleteness and the Halting problem preclude an unbreakable system. Any certificate that code is bug free, secure, has no back doors, halts, etc. is doomed to fail, in general, for systems that are complex enough.

The situation is, theoretically, much worse, where one can even embed backdoors into the compilers themselves and then have the backdoor propagate undetected in perpetuity . This has been known for a long time and Ken Thompson had a famous article "Reflections on Trusting Trust" (1984) [0] that talks about this issue.

In practice, this tends to be very difficult. Extra functionality introduces bloat and complexity. Obfuscation needs to be de-obfuscated by some method, usually known methods. Open source, many eyes, re-implementation and observation and mitigation tend to catch these issues.

Vulnerabilities (in my opinion) tend to get introduced from unintended behavior (bugs) and, very rarely, through directed premeditated malicious action (supply chain attacks, say). Unintended behavior can be exploited by attackers but wasn't introduced with the intent of obfuscation, whereas malice action tends to try and hide themselves.

There's no silver bullet. The best we can do is try to create systems that have certificates of security where we can (for, say restricted computation models) and create empirical tests and responses for the rest. For example, SQL is Turning machine equivalent, so the way around running into the halting problem is to have a timeout that will fail when triggered and leave it up to the user to create a query that runs under time constraints.

You come from biology, so you should know better than most that our body is under constant assault from foreign invaders. Some viruses come in and rewrite code that can then sneak past the immune response, replicate themselves, etc. Our body has evolved a bunch of empirical responses to such malicious action. "Cell not communicating properly?" kill it. "We're under attack?" chop up the invader to get identifying signatures for a counterattack. I'm sure you know better than me all the intricate and complicated ways our immune system handles attacks. It's not perfect, but it does pretty well, considering.

I think there is a contingent of computer scientists and software developers that we should only be using restricted computation models for critical software, so that we can reason about what the software does. This effectively sidesteps Godel's incompleteness/Halting problem because it's not general computing. My feeling is those are, as well, doomed to failure as Turing machine completeness is the norm and very hard to avoid and any system that's so crippled as to not be Turing machine equivalent has limited use.

[0] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...