News for hackers. Stuff that matters.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by londons_explore on Saturday October 28, 2023 @06:03AM (#38048541)
Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer to have the biggest chance at state-of-the-art results.

In terms of FLOPS, Google is rumoured to be by far the biggest, since they have lots of their own silicon, while most other people are limited by what NVidia can make or smallish amounts of custom silicon.

If Google can't turn that into a commercial product, the researchers don't really care.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @06:17AM (#38048614)
> and have access to far more compute than anyone else can offer to have the biggest chance at state of the art results

What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good.

On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997).

And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today.

Hinton invented Dropout (2014) and backpropagation (1986).

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by tempusalaria on Saturday October 28, 2023 @06:38AM (#38048699)
OpenAI didn’t even exist when Adam was invented!

OpenAI has made 1 major contribution - the auto regressive decoder

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by londons_explore on Saturday October 28, 2023 @07:10AM (#38048874)
Plus all the things they haven't yet published... there must be something, because nobody has yet come close to the performance of GPT-4 8 months later, despite many people trying.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @07:22AM (#38048943)
I believe they are using much more synthetic data than others. That's the secret sauce besides their larger training set and model. Synthetic data makes the model more consistent.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by washadjeffmad on Saturday October 28, 2023 @09:33AM (#38049656)
For all of the novel interest in the tech and service side, I can't understand why so many overlook the value of the datasets. I've been promoting them as potential products to license, similar to textbook publishing, and a bit reminiscent of how Google used its search appliances. Automating methods of creation will be an immensely valuable service.

I've already introduced methods of instruct formatting within our org, because as the architectures change, what we learn from performing the work to organize and account for our internal knowledge will continuously pay off.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by tempusalaria on Saturday October 28, 2023 @07:20AM (#38048931)
The likely architecture of GPT-4 was invented at Google - MoE. OpenAI poached the team who developed it.

OpenAI is ahead on the RL data side. To me that’s the likely biggest advantage they have in GPT-4.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by danenania on Saturday October 28, 2023 @10:55PM (#38055491)
Wouldn’t Google be able to catch up easily if that was really OpenAI’s only advantage? Google has a lot more resources to throw at RL. It seems like there must be some other secret sauce for OpenAI to continue maintaining such a huge lead.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by og_kalu on Saturday October 28, 2023 @10:00AM (#38049860)
Nobody has tried to come close to the performance of 4 and failed.

Palm-2 was never intended to match 4 (Gemini is). Llama 2 was never intended to match 4 either. Same with Claude 2. Nobody has matched GPT-4's training compute and failed to reach it.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @07:18AM (#38048922)
True, the paper is dated Jan 2015 and company was founded in Dec. But if you read the paper it is signed:

> Diederik P. Kingma* University of Amsterdam, OpenAI [email protected]

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by blovescoffee on Saturday October 28, 2023 @08:05AM (#38049178)
What? CLIP? Whisper? Dota 2 RL was big at the time, they made notable contributions to deep rl
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by londons_explore on Saturday October 28, 2023 @06:25AM (#38048652)
They're good at research.

They're bad at turning the research into a product.

Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by mrkramer on Saturday October 28, 2023 @08:43AM (#38049369)
>They're good at research. They're bad at turning the research into a product.

Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by logicchains on Saturday October 28, 2023 @07:28AM (#38048975)
>On pure research they invented the transformer in 2017 (Vaswani)

Transformers were invented in the early 90s: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... Google were just the first with the compute to scale them up.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by blovescoffee on Saturday October 28, 2023 @01:21PM (#38051642)
fwiw Schmidhuber is famous for claiming to have come up with virtually every architecture that's now useful/important.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by blovescoffee on Saturday October 28, 2023 @08:01AM (#38049151)
Google researchers publish leading/important research monthly. They publish impressive work in vision, nlp, and RL. Have you tried reading papers or following research out of google?
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by tester756 on Saturday October 28, 2023 @09:15AM (#38049543)
>translation the same

What? between what languages?

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by garblegarble on Saturday October 28, 2023 @06:09AM (#38048576)
>Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer.

I think that's core to what I'm getting at - I don't see how Google continues to let them publish core research at this point given how much of a lead they've given their competitors (especially combined with their own inability to execute)

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by leereeves on Saturday October 28, 2023 @06:18AM (#38048618)
If Google didn't let them publish, the researchers would leave for sure.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by aix1 on Saturday October 28, 2023 @08:58AM (#38049441)
> I don't see how Google continues to let them publish core research at this point

From what I hear from Google DeepMind researchers, the current policies are making it very difficult to publish NLP research. The barrier to disclosing research advances to the wider community appears much higher than it used to be.