News for hackers. Stuff that matters.
Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @02:59PM (#35346113)
GPT-2 was four years ago. The pace has indeed been enourmous. The fact that the general public has only been paying attention recently really doesn't mean much
Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @03:00PM (#35346135)
Meh. The difference between GPT-2 and GPT-4 is not as revolutionary as you're making it seem. The ability to process images and text as input is impressive, and also an evolutionary combination of things that were happening separately before.
Re:Keeping up with the overwhelming pace of AI innovation by crop_rotation on Tuesday March 28, 2023 @03:02PM (#35346171)
It is revolutionary for many. GPT2 was just another "looks cool" thing, GPT4 is just so jaw dropping that I am scared what GPT5 would look like.
Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @03:08PM (#35346248)
Yeah lol. I didn't want to argue further because hey it's his/her opinion.

But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me.

Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @04:42PM (#35347524)
Yours is also just an opinion. But having a baseline understanding of what came before is important to having an informed opinion. To that end: GPT is not the first tech to combine "NLP" into a single model. For example, from 2019 [1].

This stuff is evolutionary.

[1] https://pub.towardsai.net/unified-language-model-pre-trainin...

Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @07:02PM (#35349313)
You linked a paper that came months after the release of GPT-2. It is also a modified transformer, the basis of most LLMs today and an architecture that was released only two years prior.

and this is without mentioning that NLP is far from the only things LLMs do/excel at.

Frankly, nothing you have said so far has made me think your opinion is more informed than mine.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @07:46PM (#35349786)
No. I linked to a blog article which mentions many different papers, many of which were from 2018 or before. ELMO, BERT, UNILM...all examples of multi-target NLP models that pre-dated GPT.

I didn't dig this up because I thought the date on the webpage was some kind of rebuttal. I picked the first article I saw that mentioned models I knew about, with dates.

> It is also a modified transformer

Well, yes. Exactly. We're in the echo of that work, which was the core advancement that happened circa 2017. As I keep saying, it only seems revolutionary if you haven't been paying attention to the last 5 years.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Tuesday March 28, 2023 @03:33PM (#35346592)
I'm sorry but I find this take to be absurd. Across every metric GPT3 was a massive update past GPT2 and GPT4 soundly beats GPT3 by similar margins. Beyond OpenAI, AlphaGo was a massive milestone, people thought it would take decades until AI go was at GM levels, AlphaZero blitzed AlphaGo and MuZero blitzes both of them. AlphaFold was also a massive breakthough in protein folding.

Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.

Re:Keeping up with the overwhelming pace of AI innovation by version_five on Tuesday March 28, 2023 @03:41PM (#35346703)
Lol
Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @05:28PM (#35348171)
> AlphaFold was also a massive breakthough in protein folding.

My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies.

The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough".

For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.

> Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant

Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Wednesday March 29, 2023 @11:40AM (#35358825)
You're making a different argument to what you started the thread with.

> Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it.

>ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that

>The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap

>Are these large transformers a dramatic improvement? Absolutely. No question.

>Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.

Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday April 04, 2023 @08:45PM (#35447786)
> You're making a different argument to what you started the thread with.

No, I'm not. The field has been making gradual improvement for a long time, with transformers (ca 2017) being a notable jump forward. Research is always a form of punctuated equilibrium.

> Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.

Nor did I. I said that these models have been advancing gradually for a long time. The fact that particular steps can be characterized as "jumps" when viewed in isolation is not a rebuttal.

Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @03:02PM (#35346166)
If you say so
Re:Keeping up with the overwhelming pace of AI innovation by rvz on Tuesday March 28, 2023 @03:07PM (#35346231)
Absolutely. This.

This 'revolution' is quite fuelled by clever marketing and hype generated by Microsoft attempting to overthrow Google crown for better search. Unfortunately, that use-case has fallen flat on its face.

With hallucinations, bullshitting and regurgitating falsehoods and it continues to be unable to reason transparently and like all neural networks are just black-boxes with limited explanation of it's decisions, still requiring humans to review the output just like 'AI systems' still requiring humans to sit behind the wheel of a so-called robo-taxi 'AI' driver. Sounds like typical AI snake-oil.

The only 'safe' use-case for LLMs or even GPT-3 is 'summarization'. Everything else is complete hype; devoid of any reason to even justify the euphoric mania of one section of 'AI'.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Tuesday March 28, 2023 @03:35PM (#35346613)
I really doubt you understand LLMs at all if you think all they are good for is summarization. You will be shouting at your wall while the world changes outside.
Re:Keeping up with the overwhelming pace of AI innovation by rvz on Tuesday March 28, 2023 @11:23PM (#35351934)
> I really doubt you understand LLMs at all if you think all they are good for is summarization.

A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted.

Summarization of existing text is the only safe killer use-case for LLMs than the rest of the so-called 'applications' and the hype of applying it to everything, where it makes little sense other than attempting to jump on another hype train.

I predict that the majority of these ChatGPT-enabled startups will be closing down in the long run, due to no product market fit, easily running out of money or another bigger company over-taking them.

Re:Keeping up with the overwhelming pace of AI innovation by lostmsu on Wednesday March 29, 2023 @07:16AM (#35355503)
There are samples all around, including it being able to solve newly generated logical puzzles. There was quite a bit about variations of river crossing.
Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Wednesday March 29, 2023 @12:38PM (#35359713)
>A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted.

What point, that LLMs are good at nothing other than summarization? That's not a claim arrived at via logic or evidence, it's a poorly thought out opinion that happens to be wrong. There's nothing to refute.