maoberlehner

To some extent, but for the most part, they're paying to shift responsibility to somebody else.

The people who buy software are not the ones who pay the bill. If you hire a contractor to build custom-tailored software:

- If it is buggy or lacking, you'll be blamed.

- If it gets hacked, you'll be fired.

- If it works well and saves the company millions, you get a pat on the back.

The incentives are off. Why on earth would you not go with the established provider?

> Also, humans are not deterministic either.

Indeed, and that's why so few dare to migrate them, and so many who do fail or blow the budget many times over.

I think that was the implicit point of the comment: don't expect that with AI, suddenly we can convert all those COBOL apps with a single prompt.

Yes. And I know I can’t see it and don’t pretend I can, and that it in fact is green.

So LLMs don’t know the alphabet and its rules?

Your explanation perfectly captures another big differences between human / mammal intelligence and LLM intelligence: A child can make mistakes and (few shot) learn. A LLM can’t.

And even a child struggling with spelling won’t make a mistake like the one I have described. It will spell things wrong and not even catch the spelling mistake. But it won’t pretend and insist there is a mistake where there isn’t (okay, maybe it will, but only to troll you).

Maybe talking about “real” intelligence was not precise enough and it’s better to talk about “mammal like intelligence.”

I guess there is a chance LLMs can be trained to a level where all the questions where there is a correct answer for (basically everything that can be benchmarked) will be answered correctly. Would this be incredibly useful and make a lot of jobs obsolete? Yes. Still a very different form of intelligence.

I think we’re saying the same thing using different words. What LLMs do and what human brains do are very different things. Therefore human / biological intelligence is a different thing than LLM intelligence.

Is this phrasing something you can agree with?

Yes, actually I think it works really well for me considering that I’m not a native speaker and one thing I’m after is correcting technical correct but non-idiomatic wording.

I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that just weren’t there. Indeed, the two words each had a typo but not the one pointed out by Gemini.

The real typos were random missing letters. But the typos Gemini hallucinated were ones that are very common typos made in those words.

The only thing transformer based LLMs can ever do is _faking_ intelligence.

Which for many tasks is good enough. Even in my example above, the corrected text was flawless.

But for a whole category of tasks, LLMs without oversight will never be good enough because there simply is no real intelligence in them.

I have to say, I'm a little bit proud of myself that their approach comes close to what I consider a good method for building software with LLMs: https://markus.oberlehner.net/blog/ai-enhanced-development-b...

It seems logical, but still, my experience is the complete opposite. I think that it is an inherent problem with the technology. "Upgrade from Library v4 to Library v5" probably heavily triggers all the weights related to "Library," which most likely is a cocktail of all the training data from all the versions (makes me wonder how LLMs are even as good as they are at writing code with one version consistently - I assume because the weights related to a particular version become reinforced by every token matching the syntax of a particular version - and I guess this is the problem for those kinds of tasks).

For the (complex) upgrade use case, LLMs fail completely in my tests. I think in this case, the only way it can succeed is by searching (and finding!) for an explicit upgrade guide that describes how to upgrade from version v4 to v5 with all the edge cases relevant for your project in it.

More often than not, a guide like this just does not exist. And then you need (human?) ingenuity, not just "rename `oldMethodName` to `newMethodName` (when talking about a major upgrade like Angular 0 to Angular X or Vue 2 to Vue 3 and so on).

> A universal aspect of every human culture is a taboo against killing.

Exactly! It's taboo. It's morally wrong. It's unethical. But does it make it unnatural? It's taboo to talk about sex and what you do on the toilet, two of the most natural and human things there are.

Yet somehow ~70 million died during WWII, curious.

I agree with your points, and how I interpret "natural" has nothing to do with how often an individual does something. Many humans drive their cars every day. Does this make it a natural thing to do? I can see a definition of the word that makes it so. Still, the way I see it is that driving a car is something very unnatural to do, even if many people do it every day, but killing humans is something very natural, although most humans will never do it.

Most people will absolutely kill another human if the circumstances call for it. It's in our nature to do so.

> commonly-held idea that killing is an 'unnatural' act for humans

People killing people is one of the better-documented aspects of human history. If it's really a commonly-held idea, it's a delusional one. I think what they mean is 'unethical' rather than 'unnatural'.