News for hackers. Stuff that matters.
Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @02:33PM (#35345736)
Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it.

ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that, and so on...this has been happening for a long time now. If you ask me, the "amazing" part of ChatGPT isn't the output, it's the query understanding. The output is still (very pretty) garbage >10% of the time, and that's the danger zone...the Level 4 of autonomous thinking.

At the risk of saying something that makes me look dumb in ten years, this moment feels a lot like the "Level 5 self-driving will make driving obsolete by 2020!" panic we had circa 2015. Lots of investors and tech enthusiasts told me how stupid and shortsighted I was back then, when I said that we hadn't made as much progress as they thought we made, and how steep that remaining curve would be.

Newsflash: it's 2023. We still drive.

Re:Keeping up with the overwhelming pace of AI innovation by version_five on Tuesday March 28, 2023 @02:39PM (#35345837)
Yes thank you, I find myself in the funny position of going from what would popularly be called AI optimist to AI pessimist over a short time without changing any of my views. The fact that we've found a new mode of ML that's 95% impressive doesn't solve the problem with the other 5% which is the same thing that's happened with the other modes. We're all just still in the honeymoon phase.

I feel bad in a sense for the serious ML people working in this area, because there was never even a real period to reflect about how cool it is before having to be the grownups in the room and push back on absurd expectations.

Re:Keeping up with the overwhelming pace of AI innovation by chrbr on Tuesday March 28, 2023 @02:45PM (#35345913)
What sort of expectations are you seeing that you'd call absurd, out of curiosity? No agenda here, just also trying to sort through hype and reality.
Re:Keeping up with the overwhelming pace of AI innovation by mjr00 on Tuesday March 28, 2023 @03:28PM (#35346519)
Any expectation that we're going to see mass unemployment, or entire career paths made obsolete, is based on extremely inflated expectations of what LLMs could maybe do in the future.

It has a very similar feeling to the mid-2010s claims that Blockchain would displace bank settlements/SWIFT, completely change the supply chain, and make governments change how land/property ownership was tracked.

Re:Keeping up with the overwhelming pace of AI innovation by version_five on Tuesday March 28, 2023 @03:39PM (#35346687)
Same was true for "deep learning" 4-5 years ago. There's a demand for consulting companies to come out with "future of work" reports about how technology will impact jobs. Everyone knows it's flying-car style futurism and not serious, but the media picks it up as reality. Last cycle, there were all kinds of reposts on jobs that deep learning would make obsolete, truck drivers and radiologitst being the flagship cases. This time it's something else. It almost feels comforting to know your job is one of the predicted obsolete ones, because of how wrong these things are.
Re:Keeping up with the overwhelming pace of AI innovation by stemlord on Tuesday March 28, 2023 @09:17PM (#35350721)
That's maybe a bad comparison because blockchain was practically useful to no one whereas chatgpt, copilot, et al are legitimately starting to become useful tools.
Re:Keeping up with the overwhelming pace of AI innovation by baq on Wednesday March 29, 2023 @01:42AM (#35353098)
I’m seeing at least applied NLP positions going up in smoke… research is probably still going to be done, but as far as business is concerned, GPT ate NLP whole and didn’t even spit a single bone out.
Re:Keeping up with the overwhelming pace of AI innovation by TeMPOraL on Tuesday March 28, 2023 @02:43PM (#35345885)
> Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it.

Really? Maybe I too only recently "started paying attention to it", but it's way above my other go-to example of churn, which is JavaScript ecosystem - and that has a benefit of mostly being fashion and faux-innovation. There's nothing comparable I can think of that's been advancing as fast in recent years as AI/ML.

Re:Keeping up with the overwhelming pace of AI innovation by muyuu on Tuesday March 28, 2023 @02:46PM (#35345935)
what can be seen as overwhelming is the pace of news hitting the mainstream and that you might be expected to have seen if you're active in the industry

I'm annoyed by the large amount of minor projects that are interesting but belong in specialised news hitting front pages of large sites, and then people expecting you to have checked them out just because of it

Re:Keeping up with the overwhelming pace of AI innovation by poksn on Tuesday March 28, 2023 @02:47PM (#35345949)
Just because we're still a ways off from AGI, doesn't mean that the space isn't developing rapidly right now. I'm not sure autonomous driving is a fair comparable.

It's pretty crazy how much better AI offerings are today than they were a year ago. Non-Tech companies are now actually using it (and paying for it), which is a night and day difference from where it used to be.

Re:Keeping up with the overwhelming pace of AI innovation by electrondood on Tuesday March 28, 2023 @03:04PM (#35346195)
Advances in AI are informing vehicle autonomy. Tesla is effectively developing an LLM where the road system is the domain language. Recent innovations in NeRFs are enabling volumetric object detection with a single frame from multiple cameras, etc.
Re:Keeping up with the overwhelming pace of AI innovation by jasfi on Tuesday March 28, 2023 @02:48PM (#35345962)
ChatGPT was a tipping point.
Re:Keeping up with the overwhelming pace of AI innovation by tryitnow on Tuesday March 28, 2023 @02:55PM (#35346060)
Agreed. But I think you're referring to "AI innovation" and the author (and others) are referring to "AI product innovation," i.e. how AI is actually showing up in people's everyday life - and that pace is absolutely overwhelming.
Re:Keeping up with the overwhelming pace of AI innovation by baq on Tuesday March 28, 2023 @02:56PM (#35346084)
I don’t know. Progress may be linear for you but for mere mortals the jump from ‘useless but interesting toy’ to ‘indeed capable bullshit generator’ to ‘holy shit this thing run in a loop does my work for me’ was anything but linear.
Re:Keeping up with the overwhelming pace of AI innovation by disgruntledphd2 on Tuesday March 28, 2023 @03:35PM (#35346615)
And it turns out that the really valuable part (the ability to converse) doesn't even need that much data. That has been the most surprising part to me.
Re:Keeping up with the overwhelming pace of AI innovation by robopsychology on Tuesday March 28, 2023 @02:59PM (#35346114)
How are you running it in a loop? I thought about using GPT to generate code that passes a description & test code could be interesting
Re:Keeping up with the overwhelming pace of AI innovation by baq on Tuesday March 28, 2023 @03:35PM (#35346614)
Langchain or just feed it what it generates manually. I particularly like the ai-as-ceo publicity stunts or attempts at hooking it up to a bug tracker to manage the work and then execute it… no links on mobile but should be easy to google
Re:Keeping up with the overwhelming pace of AI innovation by robopsychology on Tuesday March 28, 2023 @05:38PM (#35348275)
Thanks!
Re:Keeping up with the overwhelming pace of AI innovation by gremlinsinc on Tuesday March 28, 2023 @06:58PM (#35349271)
I'm working on building a CEO-aaS for fun, more than profit, though that too... Something mere mortals (not langchain experts) can just input maybe some competitor info, api docs, and benefits/features and then the CEO bot basically can help begin building a company around the tech/offering/business idea etc.... Whether it's a SaaS or a marketing agency, or landscaping business -- it'll give you the best items to work on to expand your business so you can just focus on that, and not have to think about what to do to build your business.
Re:Keeping up with the overwhelming pace of AI innovation by noncoml on Tuesday March 28, 2023 @03:53PM (#35346872)
> holy shit this thing run in a loop does my work for me

I’d be very interested to hear what is your work

Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @02:59PM (#35346113)
GPT-2 was four years ago. The pace has indeed been enourmous. The fact that the general public has only been paying attention recently really doesn't mean much
Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @03:00PM (#35346135)
Meh. The difference between GPT-2 and GPT-4 is not as revolutionary as you're making it seem. The ability to process images and text as input is impressive, and also an evolutionary combination of things that were happening separately before.
Re:Keeping up with the overwhelming pace of AI innovation by crop_rotation on Tuesday March 28, 2023 @03:02PM (#35346171)
It is revolutionary for many. GPT2 was just another "looks cool" thing, GPT4 is just so jaw dropping that I am scared what GPT5 would look like.
Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @03:08PM (#35346248)
Yeah lol. I didn't want to argue further because hey it's his/her opinion.

But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me.

Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @04:42PM (#35347524)
Yours is also just an opinion. But having a baseline understanding of what came before is important to having an informed opinion. To that end: GPT is not the first tech to combine "NLP" into a single model. For example, from 2019 [1].

This stuff is evolutionary.

[1] https://pub.towardsai.net/unified-language-model-pre-trainin...

Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @07:02PM (#35349313)
You linked a paper that came months after the release of GPT-2. It is also a modified transformer, the basis of most LLMs today and an architecture that was released only two years prior.

and this is without mentioning that NLP is far from the only things LLMs do/excel at.

Frankly, nothing you have said so far has made me think your opinion is more informed than mine.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @07:46PM (#35349786)
No. I linked to a blog article which mentions many different papers, many of which were from 2018 or before. ELMO, BERT, UNILM...all examples of multi-target NLP models that pre-dated GPT.

I didn't dig this up because I thought the date on the webpage was some kind of rebuttal. I picked the first article I saw that mentioned models I knew about, with dates.

> It is also a modified transformer

Well, yes. Exactly. We're in the echo of that work, which was the core advancement that happened circa 2017. As I keep saying, it only seems revolutionary if you haven't been paying attention to the last 5 years.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Tuesday March 28, 2023 @03:33PM (#35346592)
I'm sorry but I find this take to be absurd. Across every metric GPT3 was a massive update past GPT2 and GPT4 soundly beats GPT3 by similar margins. Beyond OpenAI, AlphaGo was a massive milestone, people thought it would take decades until AI go was at GM levels, AlphaZero blitzed AlphaGo and MuZero blitzes both of them. AlphaFold was also a massive breakthough in protein folding.

Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.

Re:Keeping up with the overwhelming pace of AI innovation by version_five on Tuesday March 28, 2023 @03:41PM (#35346703)
Lol
Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @05:28PM (#35348171)
> AlphaFold was also a massive breakthough in protein folding.

My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies.

The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough".

For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.

> Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant

Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Wednesday March 29, 2023 @11:40AM (#35358825)
You're making a different argument to what you started the thread with.

> Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it.

>ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that

>The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap

>Are these large transformers a dramatic improvement? Absolutely. No question.

>Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.

Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday April 04, 2023 @08:45PM (#35447786)
> You're making a different argument to what you started the thread with.

No, I'm not. The field has been making gradual improvement for a long time, with transformers (ca 2017) being a notable jump forward. Research is always a form of punctuated equilibrium.

> Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.

Nor did I. I said that these models have been advancing gradually for a long time. The fact that particular steps can be characterized as "jumps" when viewed in isolation is not a rebuttal.

Re:Keeping up with the overwhelming pace of AI innovation by og_kalu on Tuesday March 28, 2023 @03:02PM (#35346166)
If you say so
Re:Keeping up with the overwhelming pace of AI innovation by rvz on Tuesday March 28, 2023 @03:07PM (#35346231)
Absolutely. This.

This 'revolution' is quite fuelled by clever marketing and hype generated by Microsoft attempting to overthrow Google crown for better search. Unfortunately, that use-case has fallen flat on its face.

With hallucinations, bullshitting and regurgitating falsehoods and it continues to be unable to reason transparently and like all neural networks are just black-boxes with limited explanation of it's decisions, still requiring humans to review the output just like 'AI systems' still requiring humans to sit behind the wheel of a so-called robo-taxi 'AI' driver. Sounds like typical AI snake-oil.

The only 'safe' use-case for LLMs or even GPT-3 is 'summarization'. Everything else is complete hype; devoid of any reason to even justify the euphoric mania of one section of 'AI'.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Tuesday March 28, 2023 @03:35PM (#35346613)
I really doubt you understand LLMs at all if you think all they are good for is summarization. You will be shouting at your wall while the world changes outside.
Re:Keeping up with the overwhelming pace of AI innovation by rvz on Tuesday March 28, 2023 @11:23PM (#35351934)
> I really doubt you understand LLMs at all if you think all they are good for is summarization.

A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted.

Summarization of existing text is the only safe killer use-case for LLMs than the rest of the so-called 'applications' and the hype of applying it to everything, where it makes little sense other than attempting to jump on another hype train.

I predict that the majority of these ChatGPT-enabled startups will be closing down in the long run, due to no product market fit, easily running out of money or another bigger company over-taking them.

Re:Keeping up with the overwhelming pace of AI innovation by lostmsu on Wednesday March 29, 2023 @07:16AM (#35355503)
There are samples all around, including it being able to solve newly generated logical puzzles. There was quite a bit about variations of river crossing.
Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Wednesday March 29, 2023 @12:38PM (#35359713)
>A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted.

What point, that LLMs are good at nothing other than summarization? That's not a claim arrived at via logic or evidence, it's a poorly thought out opinion that happens to be wrong. There's nothing to refute.

Re:Keeping up with the overwhelming pace of AI innovation by UncleOxidant on Tuesday March 28, 2023 @03:02PM (#35346160)
> the pace is not "overwhelming". You just started paying attention to it.

I've been paying some attention since 2015 when I worked a neural net gig. And the pace does seem to be increasing. Just in a small corner of LLMs we see LLaMa and then alpaca and then the LORA fine tuned version of the LLaMa params and then the various effects of different levels of quantization - allowing some of these models to run on things like RPis. More people are working on this stuff now meaning that the pace is going faster. Yeah, most of it is incremental, but that doesn't mean that it isn't difficult to keep up with where the best results are being found.

Re:Keeping up with the overwhelming pace of AI innovation by RC_ITR on Tuesday March 28, 2023 @03:05PM (#35346208)
It’s kind of wild to hear someone say this and it leads me to believe your expertise is outside the world of transformers.

In 2014, we were training encoder/decoder models on maybe a billion tokens, mostly limited by models architected around time steps.

Today, we are training encoder only models on trillions of tokens, mostly hindered by eminently solvable stability problems (i.e. can we make enough parallel compute for our infinitely parallel models).

Maybe that 10 years of progress feels the same as going from LSTM (1997-ish) to decoder attention (2014-ish) over 20 or so years, but it doesn’t to me.

Re:Keeping up with the overwhelming pace of AI innovation by adamsmith143 on Tuesday March 28, 2023 @03:40PM (#35346699)
Not only not in the world of transformers but I find it unlikely it's in modern ML at all. Deepmind has been putting out shocking work in Deep RL which is as impressive if not as hyped as anything from OpenAI.
Re:Keeping up with the overwhelming pace of AI innovation by lettergram on Tuesday March 28, 2023 @03:15PM (#35346341)
Having worked in the generative model space for years I agree.

People don’t realize how accurate NLP, generative and similar systems are expected to be

Re:Keeping up with the overwhelming pace of AI innovation by pessimizer on Tuesday March 28, 2023 @03:17PM (#35346365)
If you told me that LLMs should be trusted to drive cars or operate any other weaponry, I'd tell you it's never going to happen. But they're not: the jobs they'll be doing are the liberal arts/management jobs that have a tremendous fudge factor, close is more than good enough, and failure has few consequences[*]. If an LLM doesn't give me what I want, I can tweak it and try again; absolutely no sweat. Difficult to do after a fatal car accident.

edit: and the ability to construct intention from input has to make them ideal for interfaces (from an ignorant layman's pov.) Is anybody using generative AI to create interfaces for prosthetic limbs or other peripherals? Search engines offering them in order to interface with the internet seems like the least interesting thing to do with them, although (as I mentioned above) with artistic/creative output it's difficult to define failure.

Re:Keeping up with the overwhelming pace of AI innovation by imwithstoopid on Tuesday March 28, 2023 @03:28PM (#35346513)
> Newsflash: it's 2023. We still drive.

...but self-driving is still making incredible progress

lots of techies seem stuck in a mindset of "it doesn't work today, so it never will!"

Re:Keeping up with the overwhelming pace of AI innovation by blueorange8 on Tuesday March 28, 2023 @03:51PM (#35346842)
I agree with you that the "amazing" part is the query understanding.

There are two really big things I'm excited about:

1. It really understands what I am asking. What my meaning is. Gpt-4 is way way better even than gpt-3.5 at this. This is something truly different that a computer has never been able to do and (hopefully) will only get better at it. This really truly changes everything in how we interact with computers. We shouldn't be downplaying this. This is real. Today. At the same time a lot of this turning "understanding" into useful output comes at a huge amount of work on the RLHF side so we shouldn't necessarily extrapolate into the future too much just based on the current progress of openai. It's still very difficult work.

2. Everyone is now paying attention to LLM's. When the world starts paying attention to things and engineers start hacking things together, big companies start pouring money into ai, maybe Google wakes up for once, students start choosing ai as their career field, etc etc - amazing things happen. AI just went from niche to something that everyone is using and thinking about.

Re:Keeping up with the overwhelming pace of AI innovation by comte7092 on Tuesday March 28, 2023 @04:15PM (#35347167)
In popular consciousness, there seems to exist this blind optimism of “it will only get better/faster/etc from here”.

Which may be true, but as we saw with semiconductors, there’s always going to be an asymptote. And we don’t really know where that is until we hit it. Are we in 1970 semiconductor territory with LLMs, or are we in 2020 semiconductor territory? Ultimately, arguing about where we will be in ten years is somewhat futile.

Re:Keeping up with the overwhelming pace of AI innovation by surgical_fire on Tuesday March 28, 2023 @04:43PM (#35347538)
This is the kind of sober take that is sadly missing from these discussions. Reading all the people LARPing about how ChatGPT can solve everything that humans did until months ago is amusing, but gets tiresome pretty quickly.

And I say that as someone enthusiastic enough about it to pay for premium acess to the thing. Has been an amazing programming assistant, I use it as a web search on steroids.

But I think the advancements on the medium term will be only refinements over what is already here. It will be quite a while for another leap.

Re:Keeping up with the overwhelming pace of AI innovation by throwaway292939 on Tuesday March 28, 2023 @05:08PM (#35347861)
> The output is still (very pretty) garbage >10% of the time, and that's the danger zone...the Level 4 of autonomous thinking.

What use cases do you have for it? Isn't it the other way? I'd argue that the output is garbage <10% of the time, and it's those times you have to be a little careful for.

What else explains the massive adoption if it isn't providing a decent amount of utility?

Overall I agree with the sentiment though. It's like people underestimating how long it takes to complete the last 10% of a software project.

Re:Keeping up with the overwhelming pace of AI innovation by timr on Tuesday March 28, 2023 @05:49PM (#35348436)
> What use cases do you have for it?

Honestly, I don't know yet. Maybe it's a better version of those useless support chatbots that were ubiquitous during the last AI hype cycle?

Beyond that, you start to get into questions like "how much better is this technology than competing technology X, at benchmarks I care about?". But those questions don't capture the hive mind in quite the same way.

> I'd argue that the output is garbage <10% of the time, and it's those times you have to be a little careful for.

I don't know where the "bullshit threshold" is for a given task, but I've already seen articles claiming that programmers will be obsolete, citing GPT-generated code that was incorrect. So, you know...YMMV.

> What else explains the massive adoption if it isn't providing a decent amount of utility?

Hmmmm...I forgot the word. Begins with "H"...hyphen? Hippy? Hippo? Hyperbola?