News for hackers. Stuff that matters.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by Zambyte on Saturday July 06, 2024 @12:02PM (#40891267)
He claims that language models will soon run out of data to train on, and therefore stop getting better. Language models have already essentially run out of data to train on, yet they continue to get actionably better. Data quality is still a huge space for continued improvement.

Also around 46:20:

> the fundamental thing of knowing [the user] requires knowledge, which requires intellect, which GPT cannot have - it's mathematics.

I disagree with basically all of this except that knowing requires knowledge, but that's basically a tautology. Firstly, knowledge and reason do not depend on each other. Wikipedia is a highly knowledgeable system that demonstrates little to no reason (maybe the search can be considered a demonstration of reason). A calculator is a highly reasonable system with extremely primitive knowledge. Intelligent systems have knowledge and reason. Anyone who has played around with an LLM would agree that they are able to demonstrate some level of both knowledge and reason.

Also the implication that biological brains simply exist outside the domain of mathematics is... interesting.

Then they go on to talking about how you can't make small tweaks with generative video AI, like telling the actor to walk a little bit faster or slower. To that I want to highlight that ComfyUI has support for video nodes[0]. If you're not familiar with ComfyUI, just check out some tutorials. Generative AI art is a really cool skill that people downplay for usually wrong reasons.

> It can't generate new things

It's hard to take the host seriously when they say things like this...

> The technology is fundamentally unpredictable

It simply isn't. When you find a seed that is generally close to what you want, you fix the seed and tweak (because you can) from there.

[0] https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by blactuary on Saturday July 06, 2024 @12:19PM (#40891392)
"Anyone who has played around with an LLM would agree that they are able to demonstrate some level of both knowledge and reason"

Completely disagree, having used them quite a bit. They are able to regurgitate knowledge, but have no ability to reason

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by wongarsu on Saturday July 06, 2024 @01:21PM (#40891768)
To me the ability to reason is the biggest difference you can currently observe between 7B and 70B models.

People love to test this with "brain teasers". You could argue that LLMs can pattern match "what's heavier? 1kg of feathers or 2kg of steel", but there are enough examples of at the time novel puzzles that I feel confident to say that good LLMs can absolutely reason (at small scales, but it's getting better)

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by Zambyte on Saturday July 06, 2024 @12:33PM (#40891469)
What does it mean to be able to reason to you?
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by blactuary on Saturday July 06, 2024 @12:52PM (#40891594)
To think, to understand context, and to intelligently explain something. To understand WHY you are doing what you are doing. That is not what an LLM is doing.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by Zambyte on Saturday July 06, 2024 @03:27PM (#40892598)
> To think

You can think without reasoning. Recalling knowledge does not require reason.

> to understand context

Can you break this down? What does it mean to understand context?

> to intelligently explain something

LLMs can do this.

> To understand WHY you are doing what you are doing.

You can ask an LLM why they do things they have and they may reply, even with an answer that is likely correct. This again depends on how you break down "understand".

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by gaganyaan on Saturday July 06, 2024 @02:03PM (#40892028)
Humans can't explain why we do things, so I guess we're not intelligent either?
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by blactuary on Saturday July 06, 2024 @11:08PM (#40894996)
Yes we most certainly can
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by gaganyaan on Sunday July 07, 2024 @12:17AM (#40895219)
We can invent post hoc rationalizations, just like LLMs can
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by rickydroll on Saturday July 06, 2024 @09:18PM (#40894531)
Humans do lots of things about reasoning. How many times have you heard:

"We've always done it this way."

"It's tradition."

The daughter asks, "Mom, "why do you cut the end off the roast before cooking it?"

"I learned that from your grandmother," replied the mom.

The daughter turned to her grandmother and asked, "Why did you do it?"

Grandma rolled her eyes, sighed, and said, "My oven was too short for the roast."

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by blactuary on Saturday July 06, 2024 @11:10PM (#40895004)
Humans often do reasoning poorly, or choose not to use reason, or have a mistaken idea of what answer reasoning would give. They still have the ability to do so, which LLMs do not
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by rickydroll on Sunday July 07, 2024 @12:29AM (#40895281)
please give me evidence that they don't have the ability to reason. I'm not sure they can reason but I'm not ready to rule it out yet. we know how biological entities demonstrate reason, how would a non-biological entity demonstrate reason especially in the context of no body or senses
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by Zambyte on Sunday July 07, 2024 @10:37AM (#40898026)
> Do not reply to this message.

> I understand. I won't reply to that message.

> You replied. Why did you do that?

> I apologize for the confusion. You're right that I did reply, which was inconsistent with your initial instruction. This highlights an important limitation in my capabilities - I don't actually have the ability to simply not respond at all to a message. My training is designed to always generate some kind of response, even if that response is acknowledging an instruction not to respond. I should have been clearer about this limitation rather than incorrectly implying I had followed the instruction. Thank you for pointing this out, as it's a good reminder to be more precise about my capabilities and limitations.

I just had this dialog with Claude 3.5 Sonnet. Is this not a demonstration of exactly what you're describing as impossible for LLMs?

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by GaggiX on Saturday July 06, 2024 @01:03PM (#40891674)
I've used them extensively and they are very able to solve the problems that sometimes appear in my code, Claude 3.5 Sonnet in particular is able to not get stuck in an infinite loop causing the same error, and able to rewrite the entire code if it is necessary, it is surprisingly intelligent.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by somenameforme on Saturday July 06, 2024 @12:31PM (#40891462)
I think the easiest way to get the gist of what he (and I) are arguing is a thought experiment. Take an LLM back to the dawn of mankind, and train it on all of the collective knowledge of man. This would actually be impossible, because language hadn't yet been invented, so let's just imagine it could somehow read the collective mind of humanity. So cutting edge technology is poke 'em with the pointy side, science is nonexistent, and knowledge isn't that far behind.

Now run this system through our LLM for as long as you want with as much power as you want. Where is it going to take you? Nowhere really, it's still just going to be constrained to its pool of data, recombining things in primitive ways and be essentially permanently stuck in the present, until somebody gives it some new data to train on it and mix and match. Yet somehow humanity, starting from that exact same basis, would soon (relatively speaking) put a man on the Moon, unlock the secrets of the atom, discover quantum mechanics, invent/discover mathematics, and much more.

This is what is fundamentally meant by LLMs cannot create "new" knowledge. They absolutely can mix and match their pool of training statements in ways that can generate new statements. But if it's not an extremely simple 'remix' and we're in a domain where there are right and wrong answers, there's good chance it's just a nonsensical hallucination.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by wongarsu on Saturday July 06, 2024 @01:15PM (#40891735)
LLMs can absolutely create synthesize new knowledge out of existing knowledge. They can't easily do so iteratively because we haven't quite figured out memory yet. Until we figure that out you won't have an LLM discover a new theory of quantum gravity.

And even once we solve that, LLMs - just like human scientists - absolutely need new data from the outside world. Very few breakthroughs were achieved by just thinking about it long and hard, most were the the result of years of experimentation. Something LLMs simply can't do

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by somenameforme on Saturday July 06, 2024 @01:25PM (#40891800)
Recursive self improvement requires the ability to objectively select the "prime" of your own knowledge. From an LLM's perspective a hallucination and a correct answer are the same thing. It does not have any beliefs about what is true or false, because it has no concept of what it's even outputting. It's all just guess the next word. So even if the hallucination completely contradicts countless things it ostensibly "knows" to be true, it is unable to correct itself or realize that what it's outputting is unlikely to be correct.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by gaganyaan on Saturday July 06, 2024 @01:58PM (#40891993)
The same is true for a human brain in a vat. It's even true for humans historically, it took us millenia to figure out science.

When robots are powered by transformers or the like, I expect we'll see some pretty impressive results.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by somenameforme on Saturday July 06, 2024 @02:44PM (#40892294)
I don't think this is true - the main difference is internal consistency. Humans adopt a series of views and values, true or false, and tend build up from those. The accuracy doesn't really matter so much as the internal consistency, because it tends to turn out that trying to build from an invalid foundation eventually causes you to stop moving forward, and so the more factually supported values tend to win out over time.

But it's the internal consistency that really matters. LLMs have no internal consistency, because they have no way of 'adopting' a view, value, fact, or whatever else. They will randomly hallucinate things that directly contradict the overwhelming majority of their state, and then do so repeatedly in a single dialogue. If there were a human behaving in such a fashion, we would generally say they had schizophrenia or some other disorder that basically just ruins your ability to think like a human.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by gaganyaan on Saturday July 06, 2024 @03:21PM (#40892568)
Humans are infamously bad at being consistent:

https://en.m.wikipedia.org/wiki/Compartmentalization_(psycho...

The biggest mistake I see people make when criticizing LLMs is that they take the best possible modes of human thought from our best thinkers, and compare that to LLM edge cases.

Accuracy vs consistency isn't really a delineator. There's so much low-hanging fruit atm, like world models for LLMs improving drastically if you just train them longer. I'll believe the naysayers if say in 5 years GPT-4 is still near state of the art. Until then, there doesn't seem to actually be any theoretical limitations.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by xracy on Saturday July 06, 2024 @08:29PM (#40894334)
IDK, I think it's kinda important for LLMs to get the simple stuff correct in order to justify looking into the rest of the hallucinations.

Like, if you can't tell me "what day is it today?" (actual failed prompt I have seen) then there's no world where I'm going to have a more complicated follow-up conversation with you. It's just not worth my time or yours.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by somenameforme on Sunday July 07, 2024 @01:06AM (#40895386)
I agree with this, but find it a poor mode of debate. Because it results in a hole-plugging which is then called goal shifting, even though it's not - but rather a lack of precision in the goal to begin with. For example imagine it goes viral that 'wow look LLMs can't even play a half decent game of chess.' So OpenAI or whoever decides to dedicate an immense amount of post-training, hard-coding, and other fun stuff to enabling LLMs to play a decent game of chess.

But has anything changed? Well no, because it's obviously trivially possible for them to play a decent game of chess (or correctly assess the date), but it's an example of a more general issue of LLMs being generally incapable of consistently engaging in simple tasks across arbitrary domains. So you have software that can score some high thing on the LSAT or whatever, but can't competently engage in a game children can play.

The over-specialization for the sake of generating headlines and over-fitting benchmarks is, IMO, not productive. At least not in terms of creating optimal systems. If the goal is to generate money, which I guess it is, then it must be considered productive.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by xracy on Sunday July 07, 2024 @03:02PM (#40899657)
I'm not asking for someone to overfit to being able to properly answer the question of "What day is it today?" I'm giving an example of a simple question that all LLMs need to be able to answer correctly.

But like, people are on here saying that this will make scientific improvements, and until it can get past the basic stuff, it's not in the ballpark of anything more complicated. Right now, we're basically at the stage of 10 million monkeys on 10 million typewriters for 10 million hours. Like, maybe we'll get Shakespeare out of it, but are we willing to sort through all of the crap it will generate along the way, when it can't actually create a useful answer to simple questions?

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by somenameforme on Sunday July 07, 2024 @12:53AM (#40895352)
Hallucination is not an LLM "edge case." It is their normal and only state of operation. It just so happens that 'guess the next word' algorithms are capable of a reasonable frequency of success owing to the fact that a lot of our language is probably mostly redundant, making it possibly 'hallucinate' reasonable statements quite regularly.

Take what I wrote above. If you were given the context of what I have already written, then you could probably fill in most of what I wrote, to a reasonable degree of accuracy, after "It is their normal..." Because the issue is obvious and so my argument largely writes itself. To some degree even this second paragraph does.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by wongarsu on Saturday July 06, 2024 @01:53PM (#40891972)
Pause tokens [1] are directly dependent on the model recognizing whether the answer it arrived at is one it should "commit" to or whether it should abandon it and output a pause token instead.

Similarly if you ask ChatGPT about the current president of Numbitistan it will tell you that it doesn't know about a county with that name, rather than just hallucinating an answer. So it can at least in this circumstance tell the difference between knowing something and not knowing something.

1: https://arxiv.org/abs/2310.02226

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by jsnell on Saturday July 06, 2024 @01:30PM (#40891831)
Why are humans capable of doing that, but it's categorically impossible for LLMs? That's a very high level capability. What's the primitive it is built on that humans have but machines can't have?
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by GreatLdisisp88 on Wednesday July 10, 2024 @01:16PM (#40929100)
Quantum biomechanics. Human brain has built-in indeterministic pattern seeking. Computers are Turing based which 100% dependent on their programming and data inputs. All current LLMs operates on that very deterministic hardware. In order for LLMs to break thru existing glass ceiling, the hardware need to change.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by Zambyte on Saturday July 06, 2024 @12:45PM (#40891548)
You are comparing a species going from developing language to building spaceships on the order of hundreds of thousands of years, to a technology that was invented 7 years ago and only started to really be taken seriously 4 years ago. Working memory systems required for learning and iteratively developing ideas are in their infancy (on the scale of recent developments), but the recall technology (vector databases a la RAG) is quite well proven. I see no reason that a language model couldn't do iterative science in the same way humans have been with the resources available to it (APIs) using the composition of current technologies.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by somenameforme on Saturday July 06, 2024 @01:15PM (#40891736)
The current technology for LLMs is still a rather complex guess the next word algorithm. But this leaves the software with no real room to ever move beyond its training, regardless of how much time or training you give it. You could give them literally infinite processing power and they would not suddenly start developing new meaningful knowledge - it would still be little more than simple recombinations of its training dataset, until somebody gave it something new to train upon.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by xracy on Saturday July 06, 2024 @08:24PM (#40894314)
> I see no reason that a language model couldn't do iterative science in the same way humans have been with the resources available to it (APIs) using the composition of current technologies.

I think this belies a common refrain from people in non-tech feeling like people in tech tend to oversimplify their fields and claim they can "fix any problem" without any actual specific knowledge in that field. This just feels like the next iteration of that.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by Zambyte on Sunday July 07, 2024 @12:18AM (#40895225)
This seems like a lot of words to just say "you are wrong". Can you explain why? What features of the scientific method cannot be achieved through the composition of existing technologies? Feel free to be specific and use technical terms.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by xracy on Sunday July 07, 2024 @03:25PM (#40899808)
I think you're misreading this. I'm not saying "you are wrong". I'm saying there's a common trap that a lot of people in tech (and therefore a lot of people on HN fall into), where they believe that they can solve any problem from what they know of basic tech problems.

> What features of the scientific method cannot be achieved through the composition of existing technologies?

^This statement, sums up the above trap perfectly. "I know how to create a workflow, therefore, I know how to create a workflow of the scientific method, therefore I can replace a scientist with an AI robot."

But okay, how do we know what quantitatively, is a significant result? R=0.05 is the most common cut-off for this, but like, that's a number we picked, and isn't even agreed on.

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by sitkack on Saturday July 06, 2024 @02:30PM (#40892199)
> Take an LLM back to the dawn of mankind, and train it on all of the collective knowledge of man.

... and that is the start of your argument. Could it also have gnomes and robots?

Re:The A.I. Bubble is Bursting with Ed Zitron [video] by xracy on Saturday July 06, 2024 @08:30PM (#40894340)
That's not the start of an argument. It's the start of them explaining their point.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by sitkack on Saturday July 06, 2024 @09:05PM (#40894472)
Explaining a point is part of an argument. And the premise is unsound and illogical and if we had an LLM firewall their post would have been blocked until they cleaned up their reasoning.
Re:The A.I. Bubble is Bursting with Ed Zitron [video] by tim333 on Saturday July 06, 2024 @08:56PM (#40894452)
There's more to AI than LLMs though, like Alpha Fold figuring the structure of 200 million proteins for example. You can link those systems to your LLM so you can ask it what do you think of this protein and what experiment could I do on it and so on. There are many such avenues to explore. I don't see things getting stuck at a maxima for quite a while.