Re:Behind the curtain: what it feels like to work in AI righ...
by brendamn on Thursday April 06, 2023 @08:52PM
(#35476692)
> I'd say with GPT4 they probably reached the limit of how big the dataset can be
I’m curious about this too; not just on the dataset size, but also the model size. My hunch is that the rapid improvements of the underlying model by making it bigger/giving it more data will slow, and there’ll be more focus on shrinking the models/other optimisations.
Re:Behind the curtain: what it feels like to work in AI righ...
by int_19h on Friday April 07, 2023 @04:22PM
(#35486509)
I don't think we're anywhere close to the limit of sheer hardware scalability on this. Returns are diminishing, but if GPT-4 (with its 8+ k context window) is any indication, even those diminishing returns are still very worthwhile.
If anything, I wonder if the actual limit that'll be hit first will be the global manufacturing capacity for relevant hardware. Check out the stock price of NVDA since last October.
Re:Behind the curtain: what it feels like to work in AI righ...
by flangola7 on Thursday April 06, 2023 @09:03PM
(#35476759)
According to financial reports they are building a $225 million supercomputer for AI. What we can probably expect is the same dataset with even more compute ran on it.
Re:Behind the curtain: what it feels like to work in AI righ...
by theturtletalks on Thursday April 06, 2023 @09:13PM
(#35476834)
Is there a limit on how big the context size can be?
Re:Behind the curtain: what it feels like to work in AI righ...
by kickette on Thursday April 06, 2023 @09:49PM
(#35477065)
There’s no theoretical limit
Re:Behind the curtain: what it feels like to work in AI righ...
by PeterisP on Thursday April 06, 2023 @10:34PM
(#35477316)
There is a soft limit due to the computation required; the currently used model architectures are quadratic with respect to context size, so if you want ten times larger context size, that's going to need a hundred times more effort.
Re:Behind the curtain: what it feels like to work in AI righ...
by lexandstuff on Thursday April 06, 2023 @09:18PM
(#35476858)
In this interview [1] with Ilya Sutskever, he indicates that they aren't even close to tapping out of data.
Re:Behind the curtain: what it feels like to work in AI righ...
by nr2x on Thursday April 06, 2023 @11:10PM
(#35477565)
That interview is mind blowing stuff.
Re:Behind the curtain: what it feels like to work in AI righ...
by rcpt on Friday April 07, 2023 @12:20AM
(#35478002)
Yeah great interview style. It's non stop content.
Re:Behind the curtain: what it feels like to work in AI righ...
by fock on Friday April 07, 2023 @01:30AM
(#35478369)
I wonder if we saw the same video - or maybe it is just ChatGPT being "great" in the wild? I see one guy asking another guy simple questions and getting weaselwords for an answer.
Re:Behind the curtain: what it feels like to work in AI righ...
by revelio on Friday April 07, 2023 @06:47AM
(#35480227)
The interviewer and his questions is in some way more impressive than the answers, which is weird.
Re:Behind the curtain: what it feels like to work in AI righ...
by rcpt on Friday April 07, 2023 @11:49AM
(#35482946)
Yeah. He's fast and it's just question after question. I didn't realize that was something I wanted to see in interviews.
Re:Behind the curtain: what it feels like to work in AI righ...
by revelio on Friday April 07, 2023 @05:12PM
(#35487039)
Right, that's totally it. I came away thinking the interviewer seemed way sharper than the interviewee which is pretty rare. The sheer throughput and speed of interesting questions was incredible. Too bad that many of the answers were not.
Re:Behind the curtain: what it feels like to work in AI righ...
by jakeinspace on Thursday April 06, 2023 @11:13PM
(#35477594)
To be fair, if the opposite were true, it might not be wise to admit. Saturating available high quality training data is one of the few ways anyone can see OpenAI slowing down.
Re:Behind the curtain: what it feels like to work in AI righ...
by tome on Friday April 07, 2023 @06:31AM
(#35480134)
Yes, it's a bit strange. I would have thought
1. They would already be using everything they can get 2. They would easily be able to explain what they're not using, without giving away sensitive secrets.
GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?).
GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for it's dataset, and highly optimized hyperparameters.
I'd say with GPT4 they probably reached the limit of how big the dataset can be, because they are already using all the data that exists. Thus for GPT5 they'll have to scale in other ways.