News for hackers. Stuff that matters.
Re:Just Calm Down About GPT-4 Already by thatguyknows on Saturday May 20, 2023 @11:29PM (#36017779)
As mentioned, much of our discussion is rote parroting. I can usually go into any hackernews thread and roughly know what the top discussions are going to be. It's not surprising that an AI trained on a large portion of the internet would thus look human like.

If you really poke at GPT, you begin to realize it's fairly shallow. Human intelligence is like a deep well or pond, where as GPT is a vast but shallow ocean.

Making that ocean deeper is not a trivial problem that we can just throw more compute or data at. We've pretty much tapped out that depth with GPT4 and are going to need better designs.

This could only take half a decade or it could be half a century. Plenty of enterprises stagnate for decades.

Re:Just Calm Down About GPT-4 Already by p-e-w on Saturday May 20, 2023 @11:40PM (#36017815)
> Making that ocean deeper is not a trivial problem that we can just throw more compute or data at.

You can't possibly know that, given that we don't actually understand how LLMs work on a high level.

> We've pretty much tapped out that depth with GPT4

GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?

Re:Just Calm Down About GPT-4 Already by thatguyknows on Saturday May 20, 2023 @11:41PM (#36017820)
Sam Altman said it himself. He seems like a reasonable source.
Re:Just Calm Down About GPT-4 Already by steveBK123 on Saturday May 20, 2023 @11:51PM (#36017864)
Exactly, and OpenAI has been around nearly 8 years, consumed huge amount of data with tons of compute. They are just showing us the product now.

It is possible they've reached some 80/20 point and he is pretty honest about how much more extendable the current approach really is.

Would explain going to congress and asking for regulation (of their not-quite-there-yet competitors who they want a regulatory moat against).

Re:Just Calm Down About GPT-4 Already by og_kalu on Sunday May 21, 2023 @12:19AM (#36017997)
Re:Just Calm Down About GPT-4 Already by og_kalu on Sunday May 21, 2023 @12:15AM (#36017977)
Altman didn't really say that. Reading what he actually said rather than a headline, He was alluding to economical walls. He didn't say anything about diminishing returns on scaling. And if anything, the chief scientist, Ilya thinks there's a lot left to squeeze.
Re:Just Calm Down About GPT-4 Already by gmerc on Sunday May 21, 2023 @12:41AM (#36018075)
Sure Sam Altman, the lying CEO of a tech company (they all do) should be listened to on this matter but not on the part where he thinks AGI within reach using his approach. Selective hearing.
Re:Just Calm Down About GPT-4 Already by LouisSayers on Sunday May 21, 2023 @01:01AM (#36018151)
> You can't possibly know that, given that we don't actually understand how LLMs work on a high level.

It's a fair assumption to make however - basically 80/20 rule.

AI research isn't a new thing and I bet you could go back 40/50 years where they thought they were about to have a massive breakthrough to human level intelligence.

> GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?

I'm guessing from actually using it.

GPT4 is super impressive and helpful in a practical way, but having used it myself for a while now I get this feeling also. It feels a bit like "it's been fed everything we have, with all the techniques we have, now what?"

Re:Just Calm Down About GPT-4 Already by cornel_io on Sunday May 21, 2023 @05:49AM (#36019342)
There are dozens and maybe hundreds of different approaches that could theoretically get around the limitations of GPT4 that merely haven't been trained at scale yet. There is absolutely no lack of ideas in this space, including potentially revolutionary ones, but they take time and money to prove out.
Re:Just Calm Down About GPT-4 Already by LouisSayers on Sunday May 21, 2023 @07:03AM (#36019703)
I'm sure there are lots of ideas, but it doesn't mean they're any good or will necessarily transform AI to the next level.

It's going to take time to figure out what works and what doesn't.

There's a reason why Sam Altman is saying they're not training GPT5, and it's not because they think GPT4 is good enough.

Re:Just Calm Down About GPT-4 Already by galaxyLogic on Sunday May 21, 2023 @01:49AM (#36018365)
> ... we don't actually understand how LLMs work on a high level.

Are you saying that people who created ChatGPT don't understand how it works? Or that we the rest of people don't?

Re:Just Calm Down About GPT-4 Already by og_kalu on Sunday May 21, 2023 @03:34PM (#36024053)
Training a model doesn't mean you understand what the neurons actually do to influence output. Nobody knows that. That's where the black box analogies come in. We know what goes in the box and what comes out. We don't know what the box is doing to the data
Re:Just Calm Down About GPT-4 Already by atleastoptimal on Sunday May 21, 2023 @01:48AM (#36018362)
The apparent shallowness IMO is due to the lack of long term memory and limited context. In terms of depth, the depth of the human mind is fairly limited. Put any average human to task on any creative endeavor and it's surely a regurgitation of things they've seen. The high standard of true creativity that people hold LLM's to is only a capability of a small minority of humans.

I'd wager it's far more likely 5 years than 50 LLMs get to the full depths all humans are capable of. Simply compare the state of LLMs today vs 2018.

Re:Just Calm Down About GPT-4 Already by OrderlyTiamat on Sunday May 21, 2023 @02:36AM (#36018538)
> Making that ocean deeper is not a trivial problem that we can just throw more compute or data at.

I'd say this is immediately counterindicated by the available evidence. Gpt2 was hopeless for anything other than some fun languagw games like a bot replica of a subreddit or trump. 3.5 is much much bigger, and has semi competent but limited reasoning abilities.

Gpt 4 is a vast improvement over 3.5 in various reasoning tasks. Yes, a priori I would have agreed with you that this has to stop somewhere, but not anymore. I would need to see some data of post gpt4 models to believe you.