I feel bad in a sense for the serious ML people working in this area, because there was never even a real period to reflect about how cool it is before having to be the grownups in the room and push back on absurd expectations.
It has a very similar feeling to the mid-2010s claims that Blockchain would displace bank settlements/SWIFT, completely change the supply chain, and make governments change how land/property ownership was tracked.
Really? Maybe I too only recently "started paying attention to it", but it's way above my other go-to example of churn, which is JavaScript ecosystem - and that has a benefit of mostly being fashion and faux-innovation. There's nothing comparable I can think of that's been advancing as fast in recent years as AI/ML.
I'm annoyed by the large amount of minor projects that are interesting but belong in specialised news hitting front pages of large sites, and then people expecting you to have checked them out just because of it
It's pretty crazy how much better AI offerings are today than they were a year ago. Non-Tech companies are now actually using it (and paying for it), which is a night and day difference from where it used to be.
I’d be very interested to hear what is your work
But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me.
Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.
This stuff is evolutionary.
[1] https://pub.towardsai.net/unified-language-model-pre-trainin...
and this is without mentioning that NLP is far from the only things LLMs do/excel at.
Frankly, nothing you have said so far has made me think your opinion is more informed than mine.
I didn't dig this up because I thought the date on the webpage was some kind of rebuttal. I picked the first article I saw that mentioned models I knew about, with dates.
> It is also a modified transformer
Well, yes. Exactly. We're in the echo of that work, which was the core advancement that happened circa 2017. As I keep saying, it only seems revolutionary if you haven't been paying attention to the last 5 years.
Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.
My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies.
The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough".
For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.
> Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant
Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.
> Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it.
>ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that
>The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap
>Are these large transformers a dramatic improvement? Absolutely. No question.
>Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.
Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.
No, I'm not. The field has been making gradual improvement for a long time, with transformers (ca 2017) being a notable jump forward. Research is always a form of punctuated equilibrium.
> Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.
Nor did I. I said that these models have been advancing gradually for a long time. The fact that particular steps can be characterized as "jumps" when viewed in isolation is not a rebuttal.
This 'revolution' is quite fuelled by clever marketing and hype generated by Microsoft attempting to overthrow Google crown for better search. Unfortunately, that use-case has fallen flat on its face.
With hallucinations, bullshitting and regurgitating falsehoods and it continues to be unable to reason transparently and like all neural networks are just black-boxes with limited explanation of it's decisions, still requiring humans to review the output just like 'AI systems' still requiring humans to sit behind the wheel of a so-called robo-taxi 'AI' driver. Sounds like typical AI snake-oil.
The only 'safe' use-case for LLMs or even GPT-3 is 'summarization'. Everything else is complete hype; devoid of any reason to even justify the euphoric mania of one section of 'AI'.
A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted.
Summarization of existing text is the only safe killer use-case for LLMs than the rest of the so-called 'applications' and the hype of applying it to everything, where it makes little sense other than attempting to jump on another hype train.
I predict that the majority of these ChatGPT-enabled startups will be closing down in the long run, due to no product market fit, easily running out of money or another bigger company over-taking them.
What point, that LLMs are good at nothing other than summarization? That's not a claim arrived at via logic or evidence, it's a poorly thought out opinion that happens to be wrong. There's nothing to refute.
I've been paying some attention since 2015 when I worked a neural net gig. And the pace does seem to be increasing. Just in a small corner of LLMs we see LLaMa and then alpaca and then the LORA fine tuned version of the LLaMa params and then the various effects of different levels of quantization - allowing some of these models to run on things like RPis. More people are working on this stuff now meaning that the pace is going faster. Yeah, most of it is incremental, but that doesn't mean that it isn't difficult to keep up with where the best results are being found.
In 2014, we were training encoder/decoder models on maybe a billion tokens, mostly limited by models architected around time steps.
Today, we are training encoder only models on trillions of tokens, mostly hindered by eminently solvable stability problems (i.e. can we make enough parallel compute for our infinitely parallel models).
Maybe that 10 years of progress feels the same as going from LSTM (1997-ish) to decoder attention (2014-ish) over 20 or so years, but it doesn’t to me.
People don’t realize how accurate NLP, generative and similar systems are expected to be
edit: and the ability to construct intention from input has to make them ideal for interfaces (from an ignorant layman's pov.) Is anybody using generative AI to create interfaces for prosthetic limbs or other peripherals? Search engines offering them in order to interface with the internet seems like the least interesting thing to do with them, although (as I mentioned above) with artistic/creative output it's difficult to define failure.
...but self-driving is still making incredible progress
lots of techies seem stuck in a mindset of "it doesn't work today, so it never will!"
There are two really big things I'm excited about:
1. It really understands what I am asking. What my meaning is. Gpt-4 is way way better even than gpt-3.5 at this. This is something truly different that a computer has never been able to do and (hopefully) will only get better at it. This really truly changes everything in how we interact with computers. We shouldn't be downplaying this. This is real. Today. At the same time a lot of this turning "understanding" into useful output comes at a huge amount of work on the RLHF side so we shouldn't necessarily extrapolate into the future too much just based on the current progress of openai. It's still very difficult work.
2. Everyone is now paying attention to LLM's. When the world starts paying attention to things and engineers start hacking things together, big companies start pouring money into ai, maybe Google wakes up for once, students start choosing ai as their career field, etc etc - amazing things happen. AI just went from niche to something that everyone is using and thinking about.
Which may be true, but as we saw with semiconductors, there’s always going to be an asymptote. And we don’t really know where that is until we hit it. Are we in 1970 semiconductor territory with LLMs, or are we in 2020 semiconductor territory? Ultimately, arguing about where we will be in ten years is somewhat futile.
And I say that as someone enthusiastic enough about it to pay for premium acess to the thing. Has been an amazing programming assistant, I use it as a web search on steroids.
But I think the advancements on the medium term will be only refinements over what is already here. It will be quite a while for another leap.
What use cases do you have for it? Isn't it the other way? I'd argue that the output is garbage <10% of the time, and it's those times you have to be a little careful for.
What else explains the massive adoption if it isn't providing a decent amount of utility?
Overall I agree with the sentiment though. It's like people underestimating how long it takes to complete the last 10% of a software project.
Honestly, I don't know yet. Maybe it's a better version of those useless support chatbots that were ubiquitous during the last AI hype cycle?
Beyond that, you start to get into questions like "how much better is this technology than competing technology X, at benchmarks I care about?". But those questions don't capture the hive mind in quite the same way.
> I'd argue that the output is garbage <10% of the time, and it's those times you have to be a little careful for.
I don't know where the "bullshit threshold" is for a given task, but I've already seen articles claiming that programmers will be obsolete, citing GPT-generated code that was incorrect. So, you know...YMMV.
> What else explains the massive adoption if it isn't providing a decent amount of utility?
Hmmmm...I forgot the word. Begins with "H"...hyphen? Hippy? Hippo? Hyperbola?
ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that, and so on...this has been happening for a long time now. If you ask me, the "amazing" part of ChatGPT isn't the output, it's the query understanding. The output is still (very pretty) garbage >10% of the time, and that's the danger zone...the Level 4 of autonomous thinking.
At the risk of saying something that makes me look dumb in ten years, this moment feels a lot like the "Level 5 self-driving will make driving obsolete by 2020!" panic we had circa 2015. Lots of investors and tech enthusiasts told me how stupid and shortsighted I was back then, when I said that we hadn't made as much progress as they thought we made, and how steep that remaining curve would be.
Newsflash: it's 2023. We still drive.