FishInTheWater

Learn to take a hint. I'm not going to argue this on human terms because you're playing a dumb um-akshually game.

Computer reasoning systems can solve vastly more complex problems perfectly. Expert mathematicians can solve vastly more complex problems with only minimally increased errors. The ability of LLMs to solve reasoning problems completely disintegrates when the problems get more complex.

Trying to argue that LLMs are alike humans because of you can put these three into the buckets of "No mistakes" and "Some mistakes" is ridiculous.

Nobody is calling LLMs perfect reasoning machines.

Yes.

You said humans make mistakes, my point here is, humans make mistakes precisely because they stop doing reasoning and start doing blind pattern matching estimation of the answer.

The idea that you must make no mistake reasoning before you can be considered to be reasoning has no ground.

Reading comprehension.

I did not say no mistakes. I said that the failure pattern follows that of estimated guesses; Rapidly increasing errors as the size of the problem increases.

Whereas with computer reasoning, the rate of errors does not increase at all. And with (expert) humans the rate only goes up a little.

Did you even bother looking at the link?

You are missing the point.

I am not referring to literally English or any other language. I'm referring to the structure of language problems, which is vastly simpler than any moderately complex math or programming problem.

To more explicitly spell out the reason for my unimpressed-ness: They trained a pattern-repeating-machine and found that it will repeat some of their patterns, some of which were patterns trained on.

This does not demonstrate the ability to reason abstractly about new models, so I do not care.

If the model is productionized (read: dumbed down so it isn't as expensive to run), the reasoning abilities drastically decline again.

And these reasoning abilities are still around a language model, rather than around abstract models.

This is a very effective party trick for general math, whose language quite directly maps onto these abstract concepts, but there are some holes. Information about e.g. which values may be zero isn't encoded in the language, and so this approach is liable to blundering around division-by-zero issues.

If you want a particular example to toy around with, LLMs are not fond of quaternions and their conversion to other representations.

Human this, Human that. LLMs aren't humans. "My model is crap but the human brain isn't very good at this either" is irrelevant when we have machines that are not only very good at these tasks but almost perfect at them.

Humans make such mistakes precisely because they are not perfect reasoning machines. To compare LLMs to humans is not only disingenuous, but proves my point.

(And no, I will not humour you with an argument about how the amount of wrong answers is drastically lower from human mathematicians)

Jumping from that to "incapable of abstract reasoning" is silly.

They are language models. It is explicitly what they are designed to do.

If these LLMs are not, as I claim, reasoning on language rather than the abstract model of the query, then how come they fail miserably in exactly the ways you would expect where that the case?

LLMs generalize to non linguistic patterns.

Yes, congratulations, if you turn a problem into a linguistic one LLMs can deal with them. This does not in any way go against what I said about the capabilities of LLMs.

The same levels of actual abstract reasoning can be achieved on a graphing calculator running off literal potatoes.

When you ask LLMs to provide a reasoning, the actual reasoning performed is linguistic; The LLM has (is) a model about language and performs some (limited) reasoning on that model to get an output.

But that is explicitly different from reasoning about the abstract question at hand, thus the answer is mostly a guess.

The key difference to observe is that "semantic reasoners" like computer algebra or prolog, always maintain correctness within the axioms provided. They may slow down significantly as questions get more complex, but they do not start providing wrong answers. Computers are flawless mathematicians, provided they are programmed correctly.

LLMs do provide increasingly more-wrong answers as the question gets more complex. Thus we can observe that LLMs do not abstractly reason about the question and it's model.

They're only able to answer simple (relative-to-the-model's-size) straightforward reasoning questions. Which is a nice party trick, but not broadly useful.

They can however tell you how to convert that problem into steps that can be run in an algebra system.

Usually they can't do that very well either. Converting a problem from one description to another is algebraic reasoning, subject to the issues already mentioned.

What they can do is summarize general instructions and documentation, provided adequate training data was available.

They're neither trained on such problems, nor is that a goal for LLMs

Yes. But LLMs keep being pushed for tasks that heavily involve abstract reasoning, which is dangerous as they're unsuited for it. (E.g. Any code generation that isn't mere empty boilerplate.)

This is plainly wrong. The model's growing size makes it better at guessing the outcome of a reasoning task, but little to no actual reasoning is performed.

It's trivial to prove this as well, as LLMs will still fail miserably at (larger) math problems that even basic computer algebra systems will handle with ease.

No. If these things are claimed to be sources of truth, then the bar needs to be that high.

It is precisely because people don't fact-check that the bar has to be so high.

If you were using divination for that purposes then it's no issue. Harmless superstition is fine.

But things like personality tests and other pseudoscience see regular use in hiring and promotion. And that's just ridiculous, damaging for both "honest" applicants and the company, as such processes favour dishonest people.

Most long-lived divination methods are very vague. Anything providing concrete predictions is easily proven wrong and discredited, only the vague survives.

But people rarely take ambiguous answers for what they are, and instead interpret them into something more certain.

And this lets divination exploit all kinds of biases. On top of the regular old confirmation bias, whenever the interpretation turns out wrong, people don't write off the divination method, but assume they merely "interpreted it wrong" (and often, the vagueness means they can retcon an interpretation that is true), and worse yet, assume that now they're better at interpreting so next time it's going to be a correct prediction.

Observe how little the personality tests actually say, they're just as ambiguous.

Astrology, Tarot, the I Ching, or any other kind of divination all serve the same purpose: To provide certainty where there is none. To measure the unknowable.

People fear the unknown and risk, divination lets them feel like they have some certainty about the future.

Myers Briggs, DISC, and all the other "personality tests" are the same thing, for contemporary times.

They provide no actual measurement of applicants, Myers-Briggs is especially easy to cheat and dubious in science.

The benefit is that managers feel like they're taking less risk when hiring, but that is mere delusion.

For what that concerns us here: LLMs will never learn to fact-check anything. They'll blindly regurgitate the facts they have been "taught", but never consider or evaluate "the paper cited for this fact on wikipedia is a bunch of bullshit".

Any attempt to use them to produce "facts" is ultimately just folly, in the same way Google's attempt to do so with it's search engine index is.

The "objective" safety standards are often a lot more philosophical than you think. There is no "objective" truth for road design, it is a trade-off between how important you deem the safety of pedestrians and cyclists, versus the convenience and throughput of cars.

But also, just look at fields like journalism. Journalistic ethics exist because without them they kill people.

Not ratting out your anonymous sources isn't some technical requirement laid down in the physics of the universe, it's a philosophical belief.

And yes. Choosing to not take people's personal data is a philosophical belief. But the harm isn't philosophical.

They are arbitrary only in the sense that, sans a religion's God laying down ethics from on high, all ethics are arbitrary human creation.

But we care about ethics all the same. Engineers care about ethics because if they do not, they kill people. Civil engineers care because faulty work kills people. Electrical engineers care because faulty work kills people. Mechanical engineers care because faulty work kills people.

And as software engineers, we should care because our careless work kills people.

And if we do not care, the government will force our hand. And they will not listen to pleas for them to be reasonable.

Software engineers abused people's personal data, the EU's GDPR has outlawed using all but the absolute minimum personal data. Meta has been told in court by the CJEU that advertising is not an acceptable use of personal data, even if needed to pay the bills. Ad-tech is a doomed industry.

So it's your choice. Start caring about ethics. Or the government locks our field in so much regulatory gridlock that you will wish they just outlawed it entirely.

They are facts in the sense that "The Bar believes that it is unethical to lie to the court" is a fact. It is simply factually true that the legal profession holds that ethical belief.

And thus the point, anyone who seeks to join such a profession has to accept their ethics "as if" they were facts. You can't choose other professional ethics in the sense that you can choose to hold a different opinion.

One cannot hold copyright facts, but one can "copyright" a collection of facts like a search index or a map.

The issue here is that LLMs can only act in-character if the world has already been built and written, if the prompts are so pre-chewed that you may as well just write the dialogue directly and get even better results.

Take Solaire of Astora. He's an interesting NPC not because of any depth of the dialogue, but because of how well in-tune he is to the world and game itself. A true believer in the old god, a beacon of optimism in a depressed dying world, and someone who sets the tone of the co-op multiplayer to be silly and fun.

You can't get that out of an LLM.

It just never got widespread adoption because it's just not interesting, and LLMs are no different here. The dialogue is still empty, despite being deeper and more grammatically complex than previous attempts.

If every farmer in an RPG hands out the same "collect 20 bear asses" quest it doesn't matter if they all have "detailed" randomly generated backstories and can opine about the game world, real world philosophy, or the 2024 US elections.

Facebook recently got told by the CJEU that, no, they can't use people's posts to target advertisements. Even if those ads are what's paying for the platform. That you can't claim such processing as "part of the contract" unless it is absolutely necessary in the same way the post office needs an address to send a parcel.

If Facebook can't even do that, there is no way LLMs will be allowed. (And remember. The GDPR does not care if your system doesn't distribute personal data. Any kind of processing at all falls under the GDPR's requirements)

OpenAI is already being chased by the EU's privacy agencies. Right now they're in the process of asking pointed questions, things will heat up after that.

If you look at the core argument in favour of fair use, it's that "LLMs do not copy the training data", yet this is obviously false.

For Github copilot and ChatGPT examples of it reciting large sections of training data are well known. Plenty can be found on HN. It doesn't generate a new valid windows serial key on the fly, it's memorized them.

If one wants to be cynical, it's not hard to see OpenAI/etc patching in filters to remove copyrighted content from the output precisely because it's legally catastrophic for their "fair use" claim to have the model spit out copyrighted content. As this is both copyright infringement by itself, and evidence that no matter how the internals of these models work, they store some of the training data anyway.

even with formally composable languages like JavaScript, a semblance of unpredictability — akin to the "faerie logic" metaphor — still persists

And they're ridiculed for it, and as you state, we design around them or replace such systems entirely.

making continual interaction with them unavoidable

Technology is never unavoidable or "inevitable". We can choose not to use it, or when to use it.

The notion that we can pass a set of code or words through them once and expect a flawless result is simply illogical.

Yet that is what we expect when we put these systems into production use, especially when many proposed use cases are user-facing and subject to injection attacks.

Whether it be the writing of adcopy, the processing of loan applications, or generating code, mistakes in these tasks have very real consequences.

You can't though, that's the issue. Illustrative here are tokens like "SolidGoldMagikarp", but this does happen to "normal" sequences of tokens as well.

There is no filter you can build to keep out such mistakes, any set of otherwise normal tokens could trigger the model to produce wrong output.

Because of how large these models and most prompts are, even slight changes in things like attention can cascade into extremely different results.

there are definitely mechanisms and behaviors to discover that you can reason about.

It's faerie logic. The behaviours are mere trends and observations, not underlaying truth.

The faeries reward you for offering them fruit. But offer them apple which fell from the tree exactly 74 hours ago down to the second and they'll kill you. There is no way to know ahead of time which things will upset them.

The risk here is that you're fooled into believing these systems are understandable, that you know how they work, and that you'll mistakenly use them for something where the wrong results have consequences. You'll stop double-checking the output, all humans are lazy like that, and then you'll have disaster on your hands.

Whilst BASIC/JavaScript/etc are all magic incantations to a child, a child will soon figure out there's underlaying logic, and learn the ability to reason about what code does, and what certain changes will do.

With prompts, it's all faerie logic. There is nothing to learn, there are only magic incantations that change drastically if the model is updated.

Worse yet, the incantations cannot be composed. E.g. take the SQL statement "SELECT column FROM table WHERE column = [%s]". For any given string you insert here, the output is predictable. You can even know which characters would trigger an injection attack.

With prompts you cannot predict results. Any word, phrase, or sequence of characters may upset the faeries and cause the model to misbehave in who knows what way. No processing of user-input will stop injection attacks.

Whilst it's dubious to call current software development practices "engineering", it's utterly ridiculous to do so for prompt-writing.

We have solutions for tragedies of the commons like this: Professional Associations, Unions, and Guilds.

They have the ability to raise funds through their membership fees, and can then spend that money on the common good. Membership can be 'encouraged' by requiring payment for commercial use if the company/it's employees aren't part of the association.

This solution is just not very popular, partly because of aversion to having to pay a % fee of your paycheque, partly because it's arguably "taking open source proprietary".

But if proper open-source is unsustainable, professional associations may yet be better than the alternatives of governments deciding which open source gets funding, or complete commercialisation of open source software.