News for hackers. Stuff that matters.
Re:Behind the curtain: what it feels like to work in AI righ... by zone411 on Thursday April 06, 2023 @11:02PM (#35477510)
I'm quite surprised at how little progress FAANG companies have made in recent years, as I believe much of what's happening now with ChatGPT was predictable. Here's a slide from a deck for a product I was developing in 2017: https://twitter.com/LechMazur/status/1644093407357202434/pho....
Re:Behind the curtain: what it feels like to work in AI righ... by Analog24 on Friday April 07, 2023 @01:18AM (#35478299)
Scaling up an LM from 2017 would not achieve what GPT-4 does. It's nowhere near that simple. Of course companies saw the potential of natural language interfaces, there has been billions spent on it over the years and a lot of progress was made prior to ChatGPT coming along.
Re:Behind the curtain: what it feels like to work in AI righ... by zone411 on Friday April 07, 2023 @05:38AM (#35479862)
You're making incorrect assumptions. This project wasn't about scaling any existing approaches. It was original neural net research that produced excellent results without self-attention, using a new optimizer, new ideas to prevent overfitting, but with some feature engineering, etc. Scaling it up to GPT-2 size matched its performance, and very likely scaling it up to GPT-3 levels would have matched it as well. Unclear for GPT-4 since there is very little known about it. Of course, in the meantime, most of my ideas are no longer SOTA and there has be a ton of progress in GPU hardware and frameworks like PyTorch/TF.

I understand your skepticism, but you can check out my melodies project from a year ago as a current example. There is nothing matching it yet: https://www.youtube.com/playlist?list=PLoCzMRqh5SkFPG0-RIAR8....

What you're saying about these companies recognizing the potential is clearly incorrect. It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then. Why do you think Google initiated their "code red" if they saw the potential?

Re:Behind the curtain: what it feels like to work in AI righ... by Analog24 on Friday April 07, 2023 @12:43PM (#35483741)
Do you have any publications to back up your claims about your work? They seem more than a bit grandiose. If you're ideas are as novel and useful as you say then you should publish them.

And I'm sorry, but you're completely wrong about companies recognizing commercial potential. I worked on Alexa for five years, it is a far harder problem than you think. It is nowhere near as simple as "we just weren't looking at the right NN architecture or optimizer!" You're acting like it was a novel idea to think LMs would be extremely useful if the performance was better (in 2017). I'm just trying to tell you that isn't the case.

Re:Behind the curtain: what it feels like to work in AI righ... by zone411 on Friday April 07, 2023 @02:46PM (#35485382)
No, I have no plans to openly publish any of it. Some of my researcher employees have published their own stuff. I've previously written about how it was a huge commercial mistake for Google and others to openly publish their research, and they should stop. Indeed, now OpenAI has not published a meaningful GPT-4 paper, and DeepMind has also become more cautious. This mistake has cost them billions, and for what? Recruiting? Now they lost many people to OpenAI and wasted time and effort on publishing. Publishing is fine for university researchers or those looking to get hired. I did record some research ideas in an encrypted format on Twitter for posterity: https://twitter.com/LechMazur/status/1534481734598811650.

If any of the FAANG companies recognized the commercial potential and still accomplished so little, they must be entirely incompetent. When this 2017 deck was created, I had 50k LOC (fewer would be needed now using the frameworks and libraries) plus Word and Chrome plugins. The inference was still too slow and not quite feature-complete, and it was just a writing assistant with several other features in early testing, but it seems more than enough for me to know quite well how difficult is the task.

Re:Behind the curtain: what it feels like to work in AI righ... by Analog24 on Friday April 07, 2023 @03:13PM (#35485772)
The fact that you think creating a writing assistant plugged into Word is equivalent to building general purpose, always-on voice assistant tells me all I need to know.
Re:Behind the curtain: what it feels like to work in AI righ... by zone411 on Friday April 07, 2023 @09:44PM (#35489579)
What? We were talking about making a language model. I mentioned the plugins in relation to the question of commercializing. I'm very clear about what my project was and was not doing. I get that you're bitter because Alexa became a joke with how little progress was made and the struggles of Amazon in getting the top talent are well known. How much did it cost Amazon to fail like this? Is the whole team gone? Is that the billions you mentioned?
Re:Behind the curtain: what it feels like to work in AI righ... by Analog24 on Friday April 07, 2023 @11:05PM (#35490078)
"It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then". You can't even keep a coherent discussion and you are delusional about the significance of your work. You shared a slide with nothing but generic pie-in-the-sky use-cases and you act like it gives you some credibility on the subject ("let's make an AI system that can do the work of your non-professional employees!"). And to top it off you act like you've been successful here! Again, you shared nothing but a slide with generic use-cases that a 12 year old could think up. I don't know what you think you proved. Enjoy your imaginary pedestal.