News for hackers. Stuff that matters.
Re:Behind the curtain: what it feels like to work in AI righ... by sangnoir on Thursday April 06, 2023 @09:40PM (#35477016)
> Do you think OpenAI has discovered something they haven't been open about?

They have not, which makes me curious about which company gp works for because the "F" and "G" in FAANG are publicly known to already have LLMs. Not sure about Amazon, but I'm guessing they do too.

As an outsider, the amazing thing about ML/AI research is that you get a revolutionary discovery of s technique that changes everything, and a few months later another seminal paper is published. My bet is ChatGPT is not the last word in AI, and OpenAI will not have a monopoly on discoveries that improve the state of the art. They will have to contend with the fact that Google, Meta & Amazon own their own datacenters and can likely train models for cheaper[1] than what Microsoft is paying itself via their investment in OpenAI.

1. OpenAI may get a discount, but my prediction when they burn through Microsoft, they'll wnd up being "owned" by Microsoft for all intents and purposes.

Re:Behind the curtain: what it feels like to work in AI righ... by letitgo12345 on Thursday April 06, 2023 @10:45PM (#35477399)
Issue is the data moat OAI is building. They'll have hundreds of millions of high quality user interactions with ChatGPT they can use to finetune their models. What will anyone else including Google have?
Re:Behind the curtain: what it feels like to work in AI righ... by Teever on Friday April 07, 2023 @12:17AM (#35477984)
As people make chat bots with openAI and tie them into existing chat services, organizations that offer these chat services will get their hands on that kind of data too.

Discord comes to mind.

Re:Behind the curtain: what it feels like to work in AI righ... by letitgo12345 on Friday April 07, 2023 @02:12AM (#35478636)
That's possible ya. Hope it happens
Re:Behind the curtain: what it feels like to work in AI righ... by nmfisher on Friday April 07, 2023 @12:20AM (#35477996)
I'm not sure how deep that moat is. As soon as you open up the API, anyone can distil ChatGPT (or at least, some smaller part of it) by fine-tuning another model on its outputs[0].

I'm guessing that this is the #1 fear for people inside OpenAI have right now.

[0] For the record, I have zero problem with this.

Re:Behind the curtain: what it feels like to work in AI righ... by letitgo12345 on Friday April 07, 2023 @02:01AM (#35478548)
1. ToS make it hard for a commercial entity to do so. So some third parties would have to collect the data first

2. You won't be able to get the hundreds of millions or more interactions that OAI will have (both due to cost of API as well as it being not easy to figure out a good way to generate that many queries for a good multiturn conversaton). Maybe you can make up for it by querying smartly. We don't know if we can right now.

Re:Behind the curtain: what it feels like to work in AI righ... by sangnoir on Friday April 07, 2023 @06:04PM (#35487594)
> What will anyone else including Google have?

Google has been collecting user interactions since 2007 via GOOG-411, which was a precursor to the Google Assistant - I suspect Google has billions of user interactions on hand through the latter. Facebook has posts and comment, Amazon has products pages, reviews and product Q&As and all of them have billions of dollars to draw upon if they choose to buy high-quality data, or spin-up / increase teams that create and/or categorize training data.

They also have deep roster of AI researchers[1] to potentially obsolete LLMs or make fine-tuning work without access to of ChatGPT records.

1. I suspect Google alone has more AI researchers that OpenAI has employees