I don't think this is true for LLMs. Their output is not deterministic (up for discussion). Their weights and the sources thereof are mostly unknown to us. We cannot really be confident that an LLM will produce correct output based on correct input.

So other path forward could very well be LLMs as they can save lot of time with writing boilerplate code

They can't even perform basic arithmetic (which is not surprising since they operate at the syntactic level, oblivious to any semantic rules), yet people seem to think offloading more complex tasks with strict correctness requirements is a good idea. Boggles the mind tbh.

When I’m calling ‘getFirstChar’ from a library, me and the author have a good understanding of what the function does based on a shared context of common solutions in the domain we’re working in.

When you ask ChatGPT to write a function that does the same, your social contract is between you and untold billions of documents that you hope the algorithm weights correctly according to your prompt (we should probably avoid programming by hope).

You could probably get around this by training on your codebase as the corpus, but until we answer all the questions about what that entails it remains, well, questionable.

I use Cursor at work, which is basically VSCode + LLM for code generation. It's a guess and check, basically. Plenty of people look up StackOverflow answers to their problem, then verify that the answer does what they want. (Some people don't verify but those people are probably not good programmers I guess.) Well, sometimes I get the LLM to complete something, then verify the code is completed is what I would have written (and correct it if not). This saves time/typing for me in the long run even if I have to correct it at times. And I don't see anything wrong with this. I'm not programming by hope, I'm just saving time.

However ... your colleagues just do the same.

We'll see how this unfolds. As for now the industry seems to be a bit stuck at this level. Big models too expensive to train for marginal gains, smaller are getting better but doesn't help this. Until some one new idea comes in how LLMs should work, we won't see the 99.95% anyway.

I find that I don't use it as much for generating code as I do for automating tedious operations. For example, moving a bunch of repeating-yourself into a function, then converting the repeating blocks into function calls. The LLM's really good at doing that quickly without requiring me to perform dozens of copy-paste operations, or a bunch of multi-cursor-fu.

Also, I don't use it to generate large blocks of code or complicated logic.

It's not that LLMs aren't deterministic, because neither are many compilers.

It's also not that LLMs produce incorrect output, because compilers do that to, sometimes.

But when a compiler produces the wrong output, it's because either (1) there's a logic error in my code, or (2) there's a logic error in the compiler†, and I can drill down and figure out what's going on (or enlist someone to help me) to fix the problem.

Let's say I tell an LLM to write a algorithm, and it produces broken code. Why didn't my prompt work? How do I fix it? Can anyone ever actually know? And what did I learn from the experience?

---

† Or I guess there could be a hardware bug. Whatever. I'm going to blame the compiler because it needs to produce bytes that work on my silicon regardless of whether the silicon makes sense.

Sure, those collections of bits tend to do exactly the same thing when executed, but that's is in some sense a subjective evaluation.

---

Szundi said in a sibling comment that I was "completely [missing] the point on purpose" by bringing up compiler determinism. I think that's fair, but it's also why I opened my post by saying "I agree [with the parent], but I want to try to define the language better." Most compilers in use today are literally not deterministic, but they are deterministic in a different sense, which is useful as a comparison point to LLMs. Well, which sense? What is the fundamental quality that makes a compiler more predictable?

I'd like to try to find the correct words, because I don't think we have them yet.

There are 2 things at play here, one is LLM with human in the loop, in which it's just a tool for programmers to do the same thing they have been doing, and the other is LLM as black box automaton. For the former, it's not a problem that the tool is undeterministic, we are double checking the results and add our manual labour anyway. The fact that a tool can fail sometimes is an unsurprising fact of engineering.

I think the criticism in this chain of comment applies more to the latter, but even it always has values to non-tech people, just like how no-code approaches are, however shitty it looks to us software enfineers.

The dismissive glibness of your comment makes me wonder if it's worth it trying to point out the obvious error in the analogy you're making. Compilers translate, LLMs generate. They are two completely different things.

When you write a program in a high-level language and pass it to a compiler, the compiler translates your program to machine code, yes. But when you prompt an LLM to generate code, what are you translating? You can pretend that you are "translating natural language to code" but LLMs are not translators, they're generators, and what you're really doing is providing a prefix for the generated string. You can generate strings form an LLM with an empty prefix; but try asking a compiler to compile an empty program.

>> Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people.

You're referring to core rope memory:

https://en.wikipedia.org/wiki/Core_rope_memory

There is no "black box" here. Programmers created the program and handed it over to others to code it up. That's like hiring someone to type your code for you at a keyboard, following your instructions to do so. You have to stretch things very far to see this as anything like compilation.

Also, really, compilers are not black boxes. Just because most people treat them as a scary unknowable thing doesn't mean that's what they are. LLms are "black boxes" because no matter how much we peer at their weights, arrays of numerical values, there's nothing we can ... er ... glean from them. They're incomprehensible to humans. Not so the code of a compiler. Even raw binary is comprehensible, with some experience.

Is that "compiling" or "translating"? Lots of people use language to C "compilers".

LLMs are great, but they are the opposite of a compiler (in a good way)

https://ai.meta.com/research/publications/meta-large-languag...

The biggest difference to me is that we have humans that claim they can explain why compilers work the way they do. But I might as well trust someone who says the same about LLMs, because honestly I have no way to verify if they speak the truth. So I am already offloading a lot of burden of proof about the systems I work on to others. And why does this ”other” need to be a human.

1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.

2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.

3. Anything invented after you're thirty-five is against the natural order of things.”

― Douglas Adams, The Salmon of Doubt: Hitchhiking the Galaxy One Last Time

I imagine that machine learning powered coding will evolve to an even blacker box, than it is today: It will transform requirements to CPU instructions (or GPU instructions, netlists, ...). Why bother to follow those indirections, that are just convenience layers for those weak carbon units (urgh)?

Simultaneously, automation will likely lead to fewer skilled programmers in the future, because there will be fewer incentives to become one.

Together those effects could lead to a situation where we are condemned to just watch.

I think OP has a point, and is about guiding the design, then overall structure and dynamic of the code. Nobody expects to write in Assembler or not using libraries, but making a concious decision on the design.

I have met few programmers, but many coders. For them coding is a job, and generally they don’t care about overall architecture, algorithm efficiency, and code elegance is limited to syntax-coloring-themes in their editor. I respect their existence, but generally they are building on top off somobody else’s effort.

> I take it you're not using a compiler to generate machine code, then?

An LLM is much, much closer to a "black box" than a compiler ...

In fact, an LLM is pretty much a black box even to the people who created it.

In practice, compilers frequently have bugs and programmers even more frequently make use of "what the compiler actually does" rather than adhering to the language specification -- to the point where the de facto spec for many languages is "what the canonical implementation does".

And the specifications change over time.

To wit, a compiler diverging from the specification is so relatively rare that people will get angry about it and demand that it be fixed, while an LLM spewing creative nonsense is so accepted and par for the course that complaining about that fact is met with a shrug and "well, what did you expect?"

However, I don't understand your claim that an LLM acting as a programming assistant "...can’t be reasoned about or meaningfully examined/debugged."

I type something, and Copilot or whatever generates code which I can then examine directly, and choose to accept or reject. That seems much easier to reason about than what's happening inside a compiler, for example.

We have no such contract with LLMs. The comparison to compilers is highly mistaken, and feels like how the cryptocurrency folks used to compare cryptocurrency to gestures vaguely "the internet" in an attempt to appropriate legitimacy.

I take it you don't write C, C++, or any language at that level? It is very common to examine compiled programs to ensure the compiler made critical optimizations. I have done that many times, there are plenty of tools to help you do that.

Yes, it can optimize things for you, but that is not its purpose.

A big feature of compilers is to find optimizations you wouldn't think of. I tried to make the point that compiled output is typically not read by humans

1) Prove Addition of natural numbers.

2) Prove two real numbers are equal.

RNNs are only TC with infinite precision and unlimited resources, once you have finite precision they are very limited.

LLMs do not have recursion at all and can't even emulate finite automations. In fact soft attention can only emulate TC_0

Feed forward networks are effectively DAGs and with soft attention, DAGs built with AND,OR, NOT, and threshold circuits.

One of the state of the art inference in code methods is bi-abduction, probably best described here.

https://fbinfer.com/docs/separation-logic-and-bi-abduction/

But this localization makes it computationally possible, and has limits.

The qualification and frame problems, combined with the very limited computational power of transformers is another lens.

LLMs being formalized doesn't solve the problem. Fine tuning and RAG can help with domain specificity, but hallucinations are a fundamental feature of LLMs, not a bug.

Either a use case accepts the LLM failure mode (competent, confident, and inevitably wrong) or another model must be found.

Gödel showed us the limits of formalization, unless we find he was wrong, that won't change.

I had just assumed that RNNs were TC, didn't think of limitation put on by bounded precision since I assumed that any bounded precision could be compensated by growing memory module.

So, after your comment, I went literature searching and I found this: https://papers.nips.cc/paper_files/paper/2021/hash/ef452c63f...

I haven't read it yet. But if it is true, then RNN would seem to be TC

> As discussed in the paper and pointed out by the reviewer, the growing memory module is non-differentiable, and so it cannot be trained directly by SGD. We acknowledge this observation.

Two stack FSA/RNN are interesting, but as of now, not usable in practice.

Thinking about it, this depends on which differences we consider aspects of the output program, and which ones we consider trivial differences that don't count. If you say "build an RPG about dragons with a party of magic using heroes" and the LLM spits one out, you reached a level of abstraction where many choices relating to taste and feeling and atmosphere (and gameplay too) are waved aside as trivial details. You might extend the prompt to add a few more, but the whole point of creating a program this way is not to care about most of the details of the resulting experience. Those can be allowed to be generic and bland, right? Unless you care about leaving your personal touch on, say, all of them.

Maybe you can't examine what the LLM is doing, but as things get more advanced we can generate code to do things, and also have it generate executable formal proofs that the code works as advertised.

LLMs are not.

They seem that way, until you're tasked with getting a repeatable, idempotent build out of a non-trivial build system.

Not with todays GPU's, you would need to run it locally with special GPU settings or run it on your CPU to ensure it is deterministic.

This is entirely different from LLMs which are opaque even to their designers, and have unpredictable flaws and hallucinations, they are probability machines based on what data they have been exposed to, which means they are not a reliable way to generate programs.

Maybe one day we'll fix this, but the current generation is not very useful for programming because of this.

disclaimer: I know next to nothing about llms. and I'm not that interested to learn about them. just asking casually.

Because parsing and writing human language in a natural way is extremely complex.

> and was it humans who generated so many?

No, it is generated using an algorithm that tries to predict the next word in human written text using the words that comes before it. It ingests basically all the text on the internet to do this, without that much text the LLM performs horribly.

To give an analogy, a carpenter might be happy with hand tools, happy with machine tools, happy with plywood, and happy with MDF. For routine jobs they may be happy to buy pre-fabbed cabinets.

But for them to employ an apprentice (AI in this example) and outsource work to them - suddenly they are no longer really acting as a carpenter, but a kind of project manager.

edit: I agree that LLMs in their current state don't really fundamentally change the game - the point I am trying to make is that it's completely understandable that everyone has their own "stop" point. Otherwise, we'd all live in IKEA mansions.

They help me with the keyboard work, not any of the actual programming.

Whether you think LLMs are spectacularly worthwhile or odious and destructive, it's crucial not to classify them as being a person instead of a software tool.

Beckett: What?

HN commenter: Well it's just like when you decided to work in French instead of English. Your art was no less because of it. Now you can use an LLM instead of French. It will be so much quicker.

LLMs provide statistically probable answers with no guarantee of correctness. It takes more time to review LLM code than it takes to write it correctly from scratch.

Using a bought brush to paint and generating a painting via a prompt are basically identical.

/s

Plus, all the parts are deterministic in this stack. Their behavior is fixed for a given input, and all the parts are interpretable, readable, verifiable and observable.

LLMs are none of that. They are stochastic probability machines, which are nondeterministic. We can't guarantee their output's correctness, and we can't fix them to guarantee correct output. They are built on tons of (unethically sourced) data, which has no correctness and quality guarantees.

Some people will love LLMs, and/or see programming as a task/burden they have to complete. Some of us love programming for the sake of it, and earn money by doing it that way, too.

So putting LLMs to the same bucket with a deterministic, task specific programming tool is both wrong, and disservice to both.

I'm also strongly against LLMs, not because of the tech, but because of how they are trained and how their shortcomings are hid and they're put forward as the "oh the savior of the woeful masses, and the silver bullet of all thy problems", and it's neither of them.

LLMs are just glorified tech demos which shows what stochastic parrots can pose as accomplishing when you feed the whole world to them.

We can (for programming, at least): run the output thru theorem prover, ensure that proof is constructive, the Curry-Howard correspondence guarantees that you can turn the output into a correct program. It doesn't guarantee that formal properties of the program correspond to the informal problem statement. But even people occasionally make such errors (a provably correct program doesn't do what we wanted it to do).

> and we can't fix them to guarantee correct output

Same thing with other systems capable of programming. That is people.

You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.

No, it doesn't mean that current LLMs are ready to replace programmers, it also doesn't mean that ML models of 2030s will not be able to.

That sounds very ambitious. Automated theorem provers are real sticklers for complete specifications in a formal language and can't parse natural language at all, but when you generate code with an LLM all you have in terms of a specification is a natural language prompt (that's your "informal problem statement"). In that case what exactly is the prover going to prove? Not the natural language prompt it can't parse!

The best you can do if you start with a natural language specification, like an LLM prompt, is to verify that the generated program compiles, i.e. that it is correct syntactically. As to semantic correctness, there, you're on your own.

Edit: I'm not really sure whether you're talking about syntactic or semantic correctness after all. Which one do you mean?

>> You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.

Note that in program synthesis we usually make a distinction between complete and incomplete specifications ("problem statements") not formal and informal. An incomplete specification may still be given in a formal language. And, for the record, yes, you can make a system that guarantees that an output program is formally consistent with an incomplete specification. There exist systems like that already. You can find a bit about this online if you search for "inductive program synthesis" but the subject is spread over a wide literature spanning many fields so it's not easy to get a clear idea about it. But, in general, it works and there are approaches that give you strong theoretical guarantees of semantic correctness.

Ah, I said "theorem prover", I should have said "proof verifier". What I meant is something like DeepMind's AlphaProof with an additional step of generating a formal specification from a natural language description of the problem. In this way we get a semantically correct program wrt the formal specification. But with current generation of LLMs we probably won't get anything for non-trivial problems (the LLM won't be able to generate a valid proof).

> Note that in program synthesis we usually make a distinction between complete and incomplete specifications

Program synthesis begins after you can coherently express an idea of what you want to do. And getting to this point might involve a ton of reasoning that will not go into a program synthesis pipeline. That's what I mean when I say an "informal problem statement": some brain dumps of half-baked ideas that doesn't even constitute an incomplete specification because they are self-contradictory (but you haven't noticed it yet).

LLMs can help here by trying to generate some specification based on a brain dump.

That's even more ambitious. From DeepMind's post on AlphaProof:

First, the problems were manually translated into formal mathematical language for our systems to understand.

https://deepmind.google/discover/blog/ai-solves-imo-problems...

DeepMind had to resort to this manual translation because LLMs are not reliable enough, and natural language is not precise enough, to declare a complete specification of a formal statement, like a program or a mathematical problem (as in AlphaProof) at least not easily.

I think you point that out in the rest of your comment but you say "the LLM won't be able to generate a valid proof" where I think you meant to say "a valid specification". Did I misunderstand?

>> Program synthesis begins after you can coherently express an idea of what you want to do.

That's not exactly right. There are two kinds of program synthesis. Deductive program synthesis is when you have a complete specification in a formal language and you basically translate it to another language, just like with a compiler. That's when you "coherently express an idea of what to do". Inductive program synthesis is when you have an incomplete specification, consisting of examples of program behaviour, usually in the form of example pairs of the inputs and outputs of the target program, but sometimes program traces (like debug logs), abstract syntax trees, program schemas (a kind of rich program template) etc.

Input-output examples are the simplest case. Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples. You don't even need to know how to write that program yourself.

What do you mean when you say "a valid specification"? There are known algorithms to check validity of a proof. How do you check that specification is valid? People are inspecting it and agree that "yes, it seems to be expressing what was intended to be expressed in the natural language", or, "no, this turn of phrase needs to be understood in a different way" or some such. Today there's no other system that can handle this kind of a task besides humans (who are fallible) and LLMs (that are much more fallible).

That is deciding that specification is valid cannot be done without human involvement. I left that part out and focused on what we can mechanistically check (that is validity of a proof).

So, no, I didn't mean "a valid specification". And, yes, I don't think that today's LLMs would be good at producing specifications that would be deemed valid by a consensus of experts.

> Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples

In a limited domain with agreed-upon rules of generalization? Sure. In general? No way. The problem of generalizing from a limited number of examples with no additional restrictions is ill-defined.

And the problem "generalize as an expert would do" is in the domain of AI.

This why you should write good tests, review the code, and don’t approve anything you don’t understand.

It’s not a reason to reject pretty good autocomplete, though.

Who said that? Everyone I've talked to warns about their shortcomings (including their creators) and even the platform where I use them has a warning plastered right under the input box saying "ChatGPT can make mistakes. Check important info."

Compilers are complex programs fraught with bugs. Modern microprocessors are hideously complex devices fraught with bugs. But at least we understand them in principle and practice.

LLMs are nonsense generators, you need a second device that can recognize correct programs to use them effectively. Only humans can do that all-important second part.

> Programming has always been about finding the next black box that is both powerful and flexible.

That's the opposite of programming. Programming is the art and science of developing reliable algorithms. You can treat programs as black boxes only after you're sure that they work correctly. Otherwise you're just engaged in a kind of cargo cult.

That the scary thing about the LLM fad: so many people seem so willing to abdicate their responsibility to actually think.

>> Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?

Hell, I design gate level logic -> map it to instructions -> use them in C for the very LLMs and can fully understand[0] every aspect of it (if it doesn't behave as expected, that is a bug) but I cannot fathom or predict how the LLMs behave when i use them even though I know their architecture and implementation.

[0] Admittedly I treat the tools I use during the process, like cad tools, compiler, as black boxes, however I know that if I want to or the need arises, I can debug/understand them.

Actually the opacity of the abstraction layer is the core of the issue. First we note that opacity is a measure in both the inner-workings of the 'box' and orthogonally (in context of LLMs) a measure of deterministic outcome.

Programming, it is asserted by some of us, is exactly the act of instructing a deterministic 'black' box.

Peer-coding with an LLM is the act of cajoling a mechanism to hopefully consistently produce the input to a sensible "blackbox". It is not programming, it is getting help. Now if the help was 100% reliable, we could discuss programming the helper.

The other day I had a vision of the future AI whisperer in the corporate setting. They wear capes of varied colors and possibly sport a wand. "It's an art you see".