News for hackers. Stuff that matters.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by garblegarble on Saturday October 28, 2023 @06:01AM (#38048529)
I can't understand the mindset of any AI researchers who stay at Google at this point.

You're there (presumably) because you want to make an impact. You gave them a massive lead with a novel approach... and Google failed to capitalise on that so spectacularly that they're now investing billions into their competitor who's miles ahead of Google, using Google's approach, all to try and head off another competitor who's miles ahead of everybody also using Google's approach

I hope that (counter to stories) they pay the researchers very well, because the best result I can see is Google shutting down what they can publish to try and stop this happening again.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @06:05AM (#38048553)
> I can't understand the mindset

I can. None of this tech will maintain Google as the chokepoint of internet and make billions for them. You can download and run a LLaMA or Mistral but can't download a Google. On-topic information has been commoditised.

Even OpenAI is in a bad spot. They owned the whole LLM mountain in 2020, now they only own the peaks. Almost the whole mountain has been conquered by open models. Their area of supremacy is shrinking by the day.

The two factors at play here are 1. ability to run LLMs locally and 2. ability to improve local LLMs with data exfiltrated from SOTA LLMs.

I foresee an era of increased privacy as a consequence of this. We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB. Bad news for ad providers, they got to move into some other profitable line of business now. Nobody got time to read garbage and ads.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by quickthrower2 on Saturday October 28, 2023 @06:37AM (#38048698)
Right now GPT4 is useful/powerful and probably anything else sucks. That may change but the mountain has real value at the top only. At least for chat used for either human assist or decision making or code generation. Categorisation, yeah you can use lesser LLMs.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by npsomaratna on Saturday October 28, 2023 @07:41AM (#38049044)
Not necessarily true anymore. As a counterexample, consider CodeLlama 33b, which is quite good (and which has replaced GPT-4 for my coding assistant needs).

OpenAI's models are likely to remain the best, but I see open models catching up and becoming "good enough." Why pay for GPT-4 when I can run a model locally for free? (Barring the initial capital cost of a GPU; and not even this if you're, say, using a Macbook)

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by revoxx3 on Saturday October 28, 2023 @08:23AM (#38049277)
What's CodeLlama 33b? I couldn't find anything regarding it online
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by jsemrau on Saturday October 28, 2023 @08:42AM (#38049355)
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by bingemaker on Saturday October 28, 2023 @09:04AM (#38049476)
Off by 1B error
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by jsemrau on Saturday October 28, 2023 @10:11AM (#38049945)
^-^
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by quickthrower2 on Sunday October 29, 2023 @05:05AM (#38056871)
Just a 1B mistake
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by JTon on Saturday October 28, 2023 @08:43AM (#38049363)
Not 100% sure but I believe Llama [1] is a LLM created by meta. Code Llama is probably one tailored as a coding assistant

[1] https://ai.meta.com/llama/

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by npsomaratna on Saturday October 28, 2023 @11:49AM (#38050745)
Whoops, sorry. 34b :)
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by hansvm on Saturday October 28, 2023 @11:51AM (#38050767)
CodeLlama spit out some of the wildest garbage when I tested it, even against GPT-3.5. Which tasks have you found it to perform well?
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by emptysongglass on Saturday October 28, 2023 @07:42AM (#38049052)
Like yes, as long as GPT-4 remains the most useful (and now they have multi-modality in Plus so they've just extended their lead) nobody will adopt anything else.

So grandparent's argument makes even less sense as search should also be fungible but it isn't when Google still provides the best search product and has since it started. If OpenAI can maintain that lead and continue to ship improvements that continue to push their peak higher, most people will not pay for anything less even when that price is free.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by HKH2 on Saturday October 28, 2023 @10:23AM (#38050035)
The GP didn't say who the value was meant to be for. There is definitely a lot of value for users, regardless of which model.

I would like LLaMa to be more local, responsive and private rather than more intelligent; it is good enough in that regard.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by jokethrowaway on Saturday October 28, 2023 @08:15AM (#38049233)
No way, for creative writing zephyr (based on mistral) is my favourite. Also no stupid limits like chatgpt

For assistance, chatgpt is the best but llama is more than adequate.

For code generation they all suck, chatgpt 4 before they nerfed it was good.

I use codellama phind locally

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by HKH2 on Saturday October 28, 2023 @09:39AM (#38049699)
I don't work in tech and I use LLaMa. It is great for giving me ideas, writing overviews which I use to make sure I have most of my bases covered (and helping me see which of my ideas are more original), getting more examples, along with other creative tasks.

Could I get my employer to pay for it if all free options disappeared? Probably, but I don't have to while they exist.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by esperent on Saturday October 28, 2023 @06:31AM (#38048668)
> We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB

I cannot wait for the day where I can create a list of blocked terms "Musk, Trump, American politics, ..." and have an intelligent LLM filter so that it doesn't matter what site I visit, whether The Verge, or The Guardian, or Reddit, all articles or posts related to these terms will be gone.

I think this has a chance to have a far higher effect than just ad tech. It's going to effect all media publishers. But I also expect them, led by Google and Facebook, to fight this tooth and nail with every dirty trick they can.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by xalava on Saturday October 28, 2023 @07:04AM (#38048849)
You may achieve this by only visiting websites or pages with tailored feeds.

The problem is that it requires a bit of self discipline. Google and Meta are probably safe then...

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by esperent on Saturday October 28, 2023 @08:26AM (#38049291)
What websites are you talking about?

I do use a feed reader already but it's not good for discovering new things. The problem is when I venture out to discover what's going on in the world I'm deluged with Musk spam. A couple of years ago it was Trump spam.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by xalava on Monday October 30, 2023 @08:56AM (#38068899)
Agreed. I use - A feed reader with multiple sources - On X, I started by muting words and accounts. I now only watch one highly curated list - Here - podcasts - Engaging with relevant people directly

I'm sometimes a bit late on new trends, but I spend more time on long forms and reinforce a meaningful network in the process.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by kubrickslair on Saturday October 28, 2023 @07:39AM (#38049032)
We started work on an open-source LLM-powered media filter at OpenLocus (https://github.com/openlocus), aiming for an alpha release by late 2023.

In the more medium term we are collaborating on improving information overload, filter bubbles & misinformation with labs at AllenAI, CMU, UPenn & Utah.

Contributions and feedback are welcome! Feel free to hit me up for an early access - email in the profile.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by eganist on Saturday October 28, 2023 @07:50AM (#38049089)
Explains why Google tried (or is trying) so hard to push WEI though. If they can't programmatically evade adblockers, might as well programmatically prevent them.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by devxpy on Saturday October 28, 2023 @07:55AM (#38049124)
I’ve just the thing for you my friend https://github.com/devxpy/anti-chatgpt
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by esperent on Saturday October 28, 2023 @08:24AM (#38049281)
That looks great, have you tested it?
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by esperent on Saturday October 28, 2023 @11:07AM (#38050366)
Oh you're the author, in that case I guess you have! Thanks for sharing.

As a suggestion, personally I'd rather the offending items just be removed entirely rather than showing "redacted".

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by kubrickslair on Saturday October 28, 2023 @09:02AM (#38049464)
This is a good start, devxpy! What other features are you planning to build over it? Hit me up if you want to collaborate.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by devxpy on Sunday October 29, 2023 @05:35AM (#38057000)
The main issue is that its very slow and expensive to browse the internet like this. The LLM will only perform well if you have it do chain of thought reasoning, and that has a latency hit because of a longer generation.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by MontgomeryPy on Saturday October 28, 2023 @10:38AM (#38050148)
Great start. Please develop a plugin I can secretly add to my Mother's browser to dial down the outrage she reads on the fly! She'll notice 'redacted' ;)
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by beepbooptheory on Saturday October 28, 2023 @08:39AM (#38049345)
This is very interesting desire to me.. Wont you end up with a weird, contrived context for everything else you do read though? Like you would be reading an article about the UN and it would say: "and then, some country put forward a new resolution." Wouldn't that, like, kinda drive you crazy?

Like I know the state of journalism is less than stellar, but patching it after the fact for each reader seems like the wrong direction. The implicit conception of "the news" in this desire reifies it into a weird kind of commodity for your personal entertainment/edification; which is precisely the conception operating today which makes it so bad!

Like, maybe, if you have psychological considerations where certain triggers are very damaging, I can kinda understand this. But if that is really the case, then just why read the news anyway? Of course you gotta read some sometimes, but in general you can read other things. There is a lifetime and a half of fiction and nonfiction to read, no GPU required!

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by theduder99 on Saturday October 28, 2023 @09:27AM (#38049616)
patching it after the fact is definitely needed. same way I use a plugin to filter out curse words because when it comes to your brain, garbage in = garbage out. patching "news" stories after the fact will never be perfect though because half the problem with "journalism" is omission of facts, not just biased half-truths or opinions.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by esperent on Saturday October 28, 2023 @11:15AM (#38050426)
I don't want to remove all mentions. If I'm reading an article about something else and it happens to mention something on my list, that's fine. I just want to remove all the top level articles and posts about these things.

It's not about being triggered by anything or trying to hide from anything. I also don't need (or even want) 100% efficacy. It's about cleaning up noise. For example, at this point I'm fully aware that Musk has turned Twitter into even more of a cesspool. I don't need any more information about that. And yet I get it, all the latest "juicy Musk gossip" any time I go near any tech sites. And it's just noise to me at this point.

Same with American politics. I'm not from the US so I'd be happy with a short monthly synopsis on what's happening there. But on the English speaking web, American politics is everywhere. It's exhausting. I want to filter it, reclaim the attention it steals from me while still being engaged with online society to a degree that I choose. And I believe that reclaiming this attention, energy, and time would allow me to engage more with subjects I do care about.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by beepbooptheory on Saturday October 28, 2023 @02:46PM (#38052336)
I see, that totally makes sense, but in that case it doesn't seem like you need a fancy AI for this, or at least it feels way overkill for what you actually want here.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by esperent on Saturday October 28, 2023 @07:24PM (#38054434)
Using an LLM allows for more flexible filters e.g. clickbait, celebrity news, people trolling in comments.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by dustingetz on Saturday October 28, 2023 @06:26AM (#38048656)
is there a strongly written essay or paper arguing this viewpoint?
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @07:58AM (#38049137)
the original: "We Have No Moat, And Neither Does OpenAI" https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by lobochrome on Saturday October 28, 2023 @06:20AM (#38048629)
Powerful vision! Thank you.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by kubrickslair on Saturday October 28, 2023 @07:46AM (#38049070)
> We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB ... Nobody got time to read garbage and ads.

Exactly, there is a paradox that is getting more extreme by the day - that social media (and the broader web) is a wellspring of knowledge, yet also a vortex of addiction, filter bubbles & lost productivity. I want one without the other.

This pushed me to start building open source at OpenLocus, contributions & feedback are welcome - details in other comment.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by dividedbyzero on Saturday October 28, 2023 @08:15AM (#38049232)
Isn't Microsoft deeply integrating those peaks into Windows and Office? That's one hell of a moat.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by peanuty1 on Saturday October 28, 2023 @08:18AM (#38049247)
Microsoft is also integrating ChatGPT into Bing.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by qiine on Saturday October 28, 2023 @08:16AM (#38049239)
"I foresee an era of increased privacy as a consequence of this. We will be filtering our browsers with LLMs"

I am contemplating feeding Firefox extensions code to AI to detect possible malicious behaviors quickly.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by staunton on Saturday October 28, 2023 @08:32AM (#38049313)
Does anyone know vision models that detect ads? I wasn't able to find any, which is kind of surprising... That sounds like an armsrace that would exist.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by petemir on Saturday October 28, 2023 @08:38AM (#38049337)
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by zo1 on Saturday October 28, 2023 @09:13AM (#38049524)
It won't work as long as you have HTTPS preventing any large scale interception and analysis of traffic. This is why Google was so quick to promote and switch to HTTPS. I bet you my left nut right now there is some random ex Googler with an NDA and interesting story to tell about this topic.

It pretty much means you have to detect the ad locally. And by then you've already lost and transferred it down at least.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by TacticalCoder on Saturday October 28, 2023 @10:35AM (#38050120)
I tried something not dissimilar, but without AI models: having webpages rendered in a real browser, but "headless". But there was really no way for the webpage to detect it was "headless": I'd render in a real browser, in an actual X session, under Xephyr. And I wouldn't be showing that Xephyr X server (so I wouldn't see what that real browser was rendering).

I'd then put gray rectangles on what where ads and only show visually that, with the grey rectangles, after having covered the ads with gray rectangles.

I just did it as some quick proof-of-concept: there are plenty different ways to do this but I liked that one. It's not dissimilar to services that renders a webpage on x different devices, without you needing to open that webpage on all these devices.

But the issue is that while it's relatively easy to get rid of ads, it's near impossible to get rid of submarine articles/blogs and it's getting harder and harder by the day to get rid of all the pointless webpages generated by ChatGPT or other LLMs that are flooding the web.

Meanwhile sticking to the sites I know (Wikipedia / HN / used car sales websites / a few forums I frequent etc.) and running a DNS locally that blocks hundreds of thousands of domains (and entire countries) is quite effective (I run unbound, which I like a lot for it's got many features and can block domains/subdomains using wildcards).

I'm pretty sure detecting and covering ads before displaying a webpage can be done but I'd say the bigger problem is submarines and overall terribly poor quality LLM generated webpages.

So basically: is it even worth it to detect ads while the web has now got a much bigger problem than ads?

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by kjkjadksj on Saturday October 28, 2023 @11:50AM (#38050755)
Absolutely its still worth blocking ads. They abuse my compute rendering client side. Off with their heads. Presumably you could use ai to sniff out putative ai content in text and flag or block that too.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by beepbooptheory on Saturday October 28, 2023 @08:54AM (#38049420)
But how can we possibly be this optimistic though? Like, time and time again there are these little glimmers of fantasy that some technology alone can alleviate the heinous/annoying parts of a mature capitalism, and every time the technology has ended up, after all is said and done, on the other side (so to speak).

Like, go back and read old mailing lists. See how everyone was so assured that computers themselves, then cryptography, then the modern internet, would create radical changes in the way the economy works, and enable avenues for greater self-determination and happiness. They all seem so naive in retrospect, knowing what we know and how it all played out.

I love that so many people are these days inspired by the llm tech to imagine a better world with them, and its important in itself to inspired like this. But precedence does not favor putting all your hopes in solely the technology itself for a better world. Hope I am wrong though!

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @04:12PM (#38052992)
It will undoubtedly be a turbulent time. All the skills and knowledge expressed in text or images is compressed into models. And these models offer back all they know to each of us according to our needs. Learning from everyone indiscriminately, serving everyone specifically, open sourcing skills and knowledge to a much deeper degree than open source had on software.

This will have social implications, it will speed up the learning process of any new technology. For example by upgrading Copilot, Github can upgrade coding practices everywhere (where it's used) to new coding standards instead of taking years, that's how fast it can go.

On another line of thinking, an image-language-action model can be used to control robots, and robot hardware is getting accessible. There is a chance for inventing self reliance automation for people, a possible solution to job loss.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by beepbooptheory on Saturday October 28, 2023 @07:51PM (#38054569)
Right, there are lots of things it could do, but the point is that these capabilities, or rather this potential, is merely necessary but not sufficient for actually following through with those nice things.

It doesn't matter what it can do when there is a truly scary amount of power and interest directed towards, e.g., making sure job loss (or unemployment in general) doesn't change too much. They'll make these things illegal before they'd let it get even close to messing something like that up!

There are already so many innovations and pieces of technology that could be helping instead of hurting. It should seem clear that the sheer capability of something could never be enough to change the state of affairs alone.

Sure the printing press was critical, but there was also a whole lot of blood shed and tumultuous times before the enlightened subject of the printing press could enjoy their new class and literacy.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by mark_l_watson on Saturday October 28, 2023 @09:33AM (#38049646)
I agree heartily with you. OpenAI may be the best, but the open models are getting good.

A weak analogy: as you might take a hybrid cloud/local compute strategy, I think it makes sense to be very flexible and use LLMs from different sources for diversity and not getting locked in. I mostly use OpenAI, but I am constantly experimenting with options. Local options are most exciting to me right now.

I usually use APIs, but OpenAI’s app that supports multi modal input images and voice conversation is impressive and points to a future mode of human computer interactions.

Exciting times!

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by londons_explore on Saturday October 28, 2023 @06:03AM (#38048541)
Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer to have the biggest chance at state-of-the-art results.

In terms of FLOPS, Google is rumoured to be by far the biggest, since they have lots of their own silicon, while most other people are limited by what NVidia can make or smallish amounts of custom silicon.

If Google can't turn that into a commercial product, the researchers don't really care.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @06:17AM (#38048614)
> and have access to far more compute than anyone else can offer to have the biggest chance at state of the art results

What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good.

On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997).

And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today.

Hinton invented Dropout (2014) and backpropagation (1986).

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by tempusalaria on Saturday October 28, 2023 @06:38AM (#38048699)
OpenAI didn’t even exist when Adam was invented!

OpenAI has made 1 major contribution - the auto regressive decoder

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by londons_explore on Saturday October 28, 2023 @07:10AM (#38048874)
Plus all the things they haven't yet published... there must be something, because nobody has yet come close to the performance of GPT-4 8 months later, despite many people trying.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @07:22AM (#38048943)
I believe they are using much more synthetic data than others. That's the secret sauce besides their larger training set and model. Synthetic data makes the model more consistent.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by washadjeffmad on Saturday October 28, 2023 @09:33AM (#38049656)
For all of the novel interest in the tech and service side, I can't understand why so many overlook the value of the datasets. I've been promoting them as potential products to license, similar to textbook publishing, and a bit reminiscent of how Google used its search appliances. Automating methods of creation will be an immensely valuable service.

I've already introduced methods of instruct formatting within our org, because as the architectures change, what we learn from performing the work to organize and account for our internal knowledge will continuously pay off.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by tempusalaria on Saturday October 28, 2023 @07:20AM (#38048931)
The likely architecture of GPT-4 was invented at Google - MoE. OpenAI poached the team who developed it.

OpenAI is ahead on the RL data side. To me that’s the likely biggest advantage they have in GPT-4.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by danenania on Saturday October 28, 2023 @10:55PM (#38055491)
Wouldn’t Google be able to catch up easily if that was really OpenAI’s only advantage? Google has a lot more resources to throw at RL. It seems like there must be some other secret sauce for OpenAI to continue maintaining such a huge lead.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by og_kalu on Saturday October 28, 2023 @10:00AM (#38049860)
Nobody has tried to come close to the performance of 4 and failed.

Palm-2 was never intended to match 4 (Gemini is). Llama 2 was never intended to match 4 either. Same with Claude 2. Nobody has matched GPT-4's training compute and failed to reach it.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by visarga on Saturday October 28, 2023 @07:18AM (#38048922)
True, the paper is dated Jan 2015 and company was founded in Dec. But if you read the paper it is signed:

> Diederik P. Kingma* University of Amsterdam, OpenAI [email protected]

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by blovescoffee on Saturday October 28, 2023 @08:05AM (#38049178)
What? CLIP? Whisper? Dota 2 RL was big at the time, they made notable contributions to deep rl
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by londons_explore on Saturday October 28, 2023 @06:25AM (#38048652)
They're good at research.

They're bad at turning the research into a product.

Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by mrkramer on Saturday October 28, 2023 @08:43AM (#38049369)
>They're good at research. They're bad at turning the research into a product.

Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by logicchains on Saturday October 28, 2023 @07:28AM (#38048975)
>On pure research they invented the transformer in 2017 (Vaswani)

Transformers were invented in the early 90s: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... Google were just the first with the compute to scale them up.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by blovescoffee on Saturday October 28, 2023 @01:21PM (#38051642)
fwiw Schmidhuber is famous for claiming to have come up with virtually every architecture that's now useful/important.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by blovescoffee on Saturday October 28, 2023 @08:01AM (#38049151)
Google researchers publish leading/important research monthly. They publish impressive work in vision, nlp, and RL. Have you tried reading papers or following research out of google?
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by tester756 on Saturday October 28, 2023 @09:15AM (#38049543)
>translation the same

What? between what languages?

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by garblegarble on Saturday October 28, 2023 @06:09AM (#38048576)
>Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer.

I think that's core to what I'm getting at - I don't see how Google continues to let them publish core research at this point given how much of a lead they've given their competitors (especially combined with their own inability to execute)

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by leereeves on Saturday October 28, 2023 @06:18AM (#38048618)
If Google didn't let them publish, the researchers would leave for sure.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by aix1 on Saturday October 28, 2023 @08:58AM (#38049441)
> I don't see how Google continues to let them publish core research at this point

From what I hear from Google DeepMind researchers, the current policies are making it very difficult to publish NLP research. The barrier to disclosing research advances to the wider community appears much higher than it used to be.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by schleck8 on Saturday October 28, 2023 @06:40AM (#38048715)
Yeah, Bard is still the worst major LLM product
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by ikekkdcjkfke on Saturday October 28, 2023 @06:11AM (#38048583)
It is also a testament to what happens when you open up powerful technology instead of hoarding
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by lobochrome on Saturday October 28, 2023 @06:21AM (#38048634)
Google AI : Xerox PARC AI : GUI
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by raverbashing on Saturday October 28, 2023 @06:52AM (#38048787)
Maybe Google doesn't see how it can turn AI into a low traction product that will only be used for renaming and turning some cosmetic improvements to justify someone's promotion

Google truly has grown complacent and whoever has a good idea feel like they can't develop it there. But if you have an idea on how Google can lose money and reputation by retiring another product they're all ears

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by toasted-subs on Saturday October 28, 2023 @07:49AM (#38049087)
Classic moment of business people not understanding how some of these technical things work.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by jeswin on Saturday October 28, 2023 @08:34AM (#38049320)
> You gave them a massive lead with a novel approach...

From everything I've read/watched, the original researchers didn't know at that point their work was going to be so impactful. In hindsight, yeah. But otherwise no.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by shawabawa3 on Saturday October 28, 2023 @08:39AM (#38049347)
> I can't understand the mindset of any AI researchers who stay at Google at this point.

Have you heard of money?

But seriously, Google have talent because they pay insane salaries like all the other big tech companies

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by hcks on Saturday October 28, 2023 @08:51AM (#38049408)
Google doesn’t pay very well, actually
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by jorlow on Saturday October 28, 2023 @08:59AM (#38049449)
They definitely pay a large number of people a large amount, but they're happy to pay less when they can.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by riku_iki on Saturday October 28, 2023 @04:11PM (#38052983)
they pay average market salaries to large number of mediocre people.
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by Frost1x on Saturday October 28, 2023 @08:44AM (#38049372)
>I can't understand the mindset of any AI researchers who stay at Google at this point. >You're there (presumably) because you want to make an impact.

I can't speak to AI researchers specifically but I can speak to research more generally. Ideally you want to focus on somewhere you can make and impact, pursue the general line of research you find interesting/promising, AND have stability to at least live (or ideally enough pay to not care).

I've known enough highly talented researchers who almost always have to sacrifice the "making a difference" and "pursuing their pathway" to large degree just to survive. Ultimately what people are paying for to be researched is what matters, be it some business R&D division's head, federal agency biased directions, of whatever philanthropic connections some organizations are able to extract decide you should be doing. It's not just medicore researchers, it's what one might consider "world class" top of their field researchers. Everyone at some point makes sacrifices to pay the bills but there's this neverending gaslighting that occurs about what researchers want to do and what areas they choose to impact or some nonsense like that. That happens in very few idealized cases. E.g., tenured professorship at highly endowed institutions where they've already invested enough time doing the stuff they don't want to game the system to get to a point they can finally pursue their own paths and so on.

If you want to focus on an impact and perform research you need to be able to self-fund that and hope the area you work isn't capital intensive (AI specifically DNN, LLM, and highly data/compute intensive approaches is for the most part pretty capital intensive, hence one reason big tech pursued them to reduce competition--you can pursue paths theoretically but ultimately unless you find approaches that aren't as capital intensive it'll be awhile before you can experimentally test and iterate approaches).

Researchers just hope to get remotely close to the area they want to work in but by in large have to chase the money and lines of research behind it if they hope to remain a researcher. Maybe Google is different but I doubt it. This happens in other industries as well with people in R&D near the tops of their field working at market leaders still caving.

If by some miracle you happen to be near the top of your field and happen to guess or have some natural deep insight to the path forward where you're fortunate enough to make a huge breakthrough, you can often just spin it out yourself or take on some investment risk. Why work for Google if you have the practical path to say AGI in your hands? Try and do it on your own or find someone who doesn't own all your IP at the end of it. You work at big corp because there's ideally some stability balance with some semi-interesting research paths to you, probably not exactly what you want but as close as you can get.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by bartwr on Saturday October 28, 2023 @08:46AM (#38049380)
I have a simple answer for you - researchers measure impact differently. If they measured it primarily by shipping products, they would work as engineers, not research scientists. As a researcher you realize impact by researching and publishing (internally or externally) and are ok with the idea that it's up to others to pick it up and turn into cool products.

Hopefully they work at the same company, but if they don't - it's not your problem to solve, but VPs and above.

But yes, it can be frustrating if you're junior and take your work personally and attribute its success to its (and even worse, your own) value. After a few years of career, you learn to look at it from a distance.

Google publishes amazing stuff, people are paid great salaries, they have a ton of fun doing it, work life balance is way better than at startups - what's not to like?

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by user8872642489 on Saturday October 28, 2023 @09:21AM (#38049579)
That's why all the researches from the Attention is all you need paper not longer work at Google. /s

Food for thought: https://www.aichat.blog/google-exodus-where-are-the-authors-...

Each of the co-authors who spoke to the FT said they wanted to discover what the toolbox they had created was capable of. “The years after the transformer were some of the most fertile years in research. It became apparent . . . the models would get smarter with more feedback,” Vaswani says. “It was too compelling not to pursue this.” [2]

[2] - https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by bartwr on Saturday October 28, 2023 @09:43AM (#38049735)
This quote says nothing about product, but about continuing research.

Most researchers don't care about products. Many don't even care about any practicality (just the scientific pursuit). How do I know? Well, I happen to work as Research Scientist. I worked at Google 5y. :) For the majority of my colleagues, "having to" support their work after it was published was a nuisance and they wanted to work on new cool stuff instead.

Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by hnfong on Saturday October 28, 2023 @11:57AM (#38050808)
I am, unironically, not sure whether your description is reflective of being a Research Scientist, or just being a Google engineer in general...
Re:Google agrees to invest up to $2B in OpenAI rival Anthropic by lern_too_spel on Saturday October 28, 2023 @12:45PM (#38051255)
> people are paid great salaries, they have a ton of fun doing it, work life balance is way better than at startups

Compared to AI startups, neither of these is true.