News for hackers. Stuff that matters.
Re:Google's First Tensor Processing Unit: Architecture by hipadev23 on Monday March 25, 2024 @10:07PM (#39823598)
How is it that Google invented the TPU and Google Research came up with the paper on LLM and NVDA and AI startup companies have captured ~100% of the value
Re:Google's First Tensor Processing Unit: Architecture by neilv on Monday March 25, 2024 @10:33PM (#39823751)
There's an old joke explanation about Xerox and PARC, about the difficulty of "pitching a 'paperless office' to a photocopier company".

In Google's case, an example analogy would be pitching making something like ChatGPT widely available, when that would disrupt revenue from search engine paid placements, and from ads on sites that people wouldn't need to visit. (So maybe someone says, better to phase it in subtly, as needed for competitiveness, but in non-disruptive ways.)

I doubt it's as simple as that, but would be funny if that was it.

Re:Google's First Tensor Processing Unit: Architecture by halflings on Monday March 25, 2024 @11:48PM (#39824102)
This (innovator's dilemma / too afraid of disrupting your own ads business model) is the most common explanation folks are giving for this, but seems to be some sort of post-rationalization of why such a large company full of competent researchers/engineers would drop the ball this hard.

My read (having seen some of this on the inside), is that it was a mix of being too worried about safety issues (OMG, the chatbot occasionally says something offensive!) and being too complacent (too comfortable with incremental changes in Search, no appetite for launching an entirely new type of product / doing something really out there). There are many ways to monetize a chatbot, OpenAI for example is raking billions in subscription fees.

Re:Google's First Tensor Processing Unit: Architecture by nemothekid on Tuesday March 26, 2024 @12:05AM (#39824186)
>There are many ways to monetize a chatbot, OpenAI for example is raking billions in subscription fees.

Compared to Google, OpenAI's billions is peanuts, while costing a fortune to generate. GPT-4 doesn't seem profitable (if it was, would they need to throttle it?)

Re:Google's First Tensor Processing Unit: Architecture by ro_sharp on Tuesday March 26, 2024 @12:34AM (#39824294)
> GPT-4 doesn't seem profitable (if it was, would they need to throttle it?)

Maybe? Hardware supply isn’t perfectly elastic

Re:Google's First Tensor Processing Unit: Architecture by nequo on Tuesday March 26, 2024 @02:20AM (#39824740)
Wouldn't Google be better able to integrate ads into a "ChatGoogle" service than OpenAI is into ChatGPT?
Re:Google's First Tensor Processing Unit: Architecture by exitheone on Tuesday March 26, 2024 @03:13AM (#39824974)
The cost per ad is still astronomically different between search ads and LLMs
Re:Google's First Tensor Processing Unit: Architecture by varjag on Tuesday March 26, 2024 @04:40AM (#39825455)
There could be an opposite avenue: ad-free Google Premium subscription with AI chat as a crown jewel. An ultimate opportunity to diversify from ad revenue.
Re:Google's First Tensor Processing Unit: Architecture by disgruntledphd2 on Tuesday March 26, 2024 @09:51AM (#39827817)
There's not enough money in it, as Google's scale.

Especially because the people who'd pay for Premium tend to be the most prized people from an advertiser perspective.

And most people won't pay, under any circumstances, but they will click on ads which make Google money.

Re:Google's First Tensor Processing Unit: Architecture by varjag on Tuesday March 26, 2024 @10:25AM (#39828238)
YouTube does it, at Google scale. And these same people do pay $20/mo for ChatGPT anyway.
Re:Google's First Tensor Processing Unit: Architecture by nemothekid on Tuesday March 26, 2024 @11:20AM (#39829024)
YouTube isn't comparable - YouTube revenue is roughly 30B/year, while Search revenue is roughly 175B/year.

Advertisers are willing to pay far more than $20/mo per user, combined with the fact that search costs way less per query than inference.

Re:Google's First Tensor Processing Unit: Architecture by nequo on Tuesday March 26, 2024 @07:49PM (#39834152)
The low operating margin of serving a GPT-4 scale model sounds like a compelling explanation for why Google stayed out of it.

But then why did Microsoft put its money behind it? Alphabet's revenue is around $300bn, and Microsoft's is around $210bn which is lower but it is the same order of magnitude.

Re:Google's First Tensor Processing Unit: Architecture by Karrot_Kream on Tuesday March 26, 2024 @01:55AM (#39824623)
Google gets much more scrutiny then smaller companies so it's understandable to be worried. Pretty much any small mistake of theirs turns into clickbait on here and the other tech news sites and you get hundreds of comments about how evil Big Tech is. Of course it's their own fault that their PR hews negative so frequently but still it's understandable why they were so shy.
Re:Google's First Tensor Processing Unit: Architecture by logicchains on Tuesday March 26, 2024 @01:58AM (#39824639)
Sydney when initially released was much less censored and the vast majority of responses online were positive, "this is hilarious/cool", not "OMG Sydney should be banned!".
Re:Google's First Tensor Processing Unit: Architecture by knowriju on Tuesday March 26, 2024 @08:12AM (#39826858)
You have clearly not heard about Tay and Galactica.
Re:Google's First Tensor Processing Unit: Architecture by Symmetry on Tuesday March 26, 2024 @08:21AM (#39826950)
It's understandable that people at Google are worried because it's likely very unpleasant to see critical articles and tweets about something you did. But that isn't really bad for Google's business in any of the ways that losing to someone on AI would be.
Re:Google's First Tensor Processing Unit: Architecture by IX-103 on Tuesday March 26, 2024 @10:35AM (#39828417)
Google is constantly being sued for nearly everything they do. They create a Chrome Incognito mode like Firefox's private browsing mode and they get sued. They start restricting App permissions on Android, sued. Adding a feature where Google maps lets you select the location of your next appointment as a destination in a single click, sued (that's leveraging your calendar monopoly to improve your map app).

Google has it's hands in so many fields that any change they make that disrupts the status-quo brings down antitrust investigations and lawsuits.

That's the reason why Firefox and Safari dropping support for 3rd party cookies gets a yawn from regulators while Google gets pinned between the CMA wanting to slow down or stop 3rd party cookies deprecation to prevent disrupting the ads market and the ICO wanting Google to drop support yesterday.

This is not about bad press or people feeling bad about news articles. Google has been hit by billion dollar fines in the past and has become hesitant to do anything.

Where smaller companies can take the "Elon Musk" route and just pay fines and settle lawsuits as just the cost of doing business, Google has become an unwieldy juggernaut unable to move out of fear of people complaining and taking another pound of flesh. To be clear, I don't agree with a strategy of ignoring inconvenient regulations, but Google's excess of caution has severely limited their ability to innovate. But given previous judgements against Google, I can't exactly say that they're wrong to do so. Even Google can only pay so many multi-billion dollar fines before they have to close shop, and I can't exactly say the world would be better off if that happened.

Re:Google's First Tensor Processing Unit: Architecture by michaelt on Tuesday March 26, 2024 @10:37AM (#39828455)
That's true for google, sure. But what about individual workers and managers at google?

You can push things forward hard, battle the many stakeholders all of whom want their thing at the top of the search results page, get a load of extra headcount to make a robust and scalable user-facing system, join an on-call rota and get called at 2am, engage in a bunch of ethically questionable behaviour skirting the border between fair use and copyright infringement, hire and manage loads of data labellers in low-income countries who get paid a pittance, battle the internal doubters who think Google Assistant shows chatbots are a joke and users don't want it, and battle the internal fearmongers who think your ML system is going to call black people monkeys, and at the end of it maybe it's great or maybe it ends up an embarrassment that gets withdrawn, like Tay.

Or you can publish some academic papers. Maybe do some work improving the automatic transcription for youtube, or translation for google translate. Finish work at 3pm on a Friday, and have plenty of time to enjoy your $400k salary.

Re:Google's First Tensor Processing Unit: Architecture by rs11 on Tuesday March 26, 2024 @02:52AM (#39824876)
Monetizing a chatbot is one thing. Beating revenues every year when you are already making 300b a year is a whole different ball game There must be tens of execs who understand this but their payout depends on keeping status quo
Re:Google's First Tensor Processing Unit: Architecture by vineyardmike on Monday March 25, 2024 @11:37PM (#39824058)
The answer is far weirder - they had a chat bot, and no one even discussed it in the context of search replacements. They didn’t want to release it because they just didn’t think it should be a product. Only after OpenAI actually disrupted search did they start releasing Gemini/Bard which takes advantage of search.
Re:Google's First Tensor Processing Unit: Architecture by ec109685 on Monday March 25, 2024 @11:46PM (#39824095)
They were afraid to release it because of unaligned output and hallucinations.

ChatGPT showed that people could still get value out of something that wasn’t perfect.

E.g. they had this in their labs: https://www.theguardian.com/technology/2022/jun/12/google-en... from July, 2022z

Re:Google's First Tensor Processing Unit: Architecture by halflings on Monday March 25, 2024 @11:50PM (#39824113)
Agree re:hallucinations/safety issues, that was likely one of the main blockers.

And here's the sad part: they had this back in 2019... see this paper released in Jan 2020: https://blog.research.google/2020/01/towards-conversational-...

Re:Google's First Tensor Processing Unit: Architecture by HarHarVeryFunny on Tuesday March 26, 2024 @02:17PM (#39831043)
LaMBDA was also briefly available for public testing, but then rapidly withdrawn due to unhinged responses.

One advantage that OpenAI had over Google was having developed RLHF as a way to "align" the model's output to be more acceptable.

Part of Google's dropping the ball at that time period (but catching up now with Gemini) may also have been just not knowing what to do with it. It certainly wasn't apparent pre-ChatGPT that there'd be any huge public demand for something like this, or that people would find so many uses for it in API form, and especially so with LaMBDA's behavioral issues.

Re:Google's First Tensor Processing Unit: Architecture by eitally on Tuesday March 26, 2024 @11:42AM (#39829273)
My take as someone who worked in Cloud, closely with the AI product teams on GTM strategy, is that it was primarily the former: Google was always extremely risk averse when it came to AI, to the point that until Andrew Moore was pushed out, Google Cloud didn't refer to anything as AI. It was ML-only, hence the BigQuery ML, Video Intelligence ML, NLP API, and so many other "ML" product names. There was strong sentiment internally that the technology wasn't mature enough to legitimately call it "AI", and that any models adequately complex to be non-trivially explainable were a no-go. Part of this was just general conservatism around product launches within Google, but it was significantly driven by EU regulation, too. Having just come off massive GDPR projects and staring down the barrel of DMA, Google didn't want to do anything that expanded the risk surface, whether it was in Cloud, Ads, Mobile or anything else.

Their hand was forced with ChatGPT was launched ... and we're seeing how that's going.

Re:Google's First Tensor Processing Unit: Architecture by tw04 on Monday March 25, 2024 @10:32PM (#39823744)
Because Google can’t focus on a product for more than 18 months if it isn’t generating several billion in PROFIT. They are punch drunk on advertising.
Re:Google's First Tensor Processing Unit: Architecture by Culonavirus on Tuesday March 26, 2024 @05:01AM (#39825609)
They're like a hyperactive dog chasing its own tail. How many projects did they create only to shut them a bit later? All because there's always some nonsense to chase. Meanwhile the AI train has left the station without them and their search is now an ad infested hot piece of garbage. Don't even get me started on their customer/dev support or how aging things like Google Translate api got absolutely KILLED by GPT-4 like apis overnight.

Google has stage 4 leadership incompetency and can't be helped. The only humane option is euthanasia.

Re:Google's First Tensor Processing Unit: Architecture by wstrange on Monday March 25, 2024 @10:23PM (#39823707)
It's far too early to suggest Google will not capture value from AI. They have plenty of opportunity to integrate AI into their products.
Re:Google's First Tensor Processing Unit: Architecture by earthnail on Tuesday March 26, 2024 @04:03AM (#39825221)
Microsoft is sooooo far ahead in this game, it’s borderline ridiculous. Google really missed an opportunity to grab major market share from MS Office.
Re:Google's First Tensor Processing Unit: Architecture by willvarfar on Tuesday March 26, 2024 @05:25AM (#39825755)
Yes, this! Google Docs is basically basic. But imagine if, years ago, Google had added built-in LLM-based auto-complete and refactoring and summation tools to documents and presentations etc, years ago...

Oh to dream.

Re:Google's First Tensor Processing Unit: Architecture by abraae on Monday March 25, 2024 @10:09PM (#39823611)
For historical precedent see Xerox Parc.
Re:Google's First Tensor Processing Unit: Architecture by readyplayernull on Monday March 25, 2024 @10:13PM (#39823639)
IBM, Intel, Apple's Newton.
Re:Google's First Tensor Processing Unit: Architecture by klodolph on Tuesday March 26, 2024 @12:09AM (#39824206)
The story I like to tell for the Newton is that it was launched before the technology was ready yet. Like the Sega Game Gear. Old video phones. All those tablets that launched before the iPad.

They’re good ideas, but they shipped a few years too early, and the technology to make them work well at a good price point wasn’t available until later. Like, the Sega Game Gear had a cool active matrix LCD screen, but it took six AA batteries and the batteries only lasted like four hours.

Re:Google's First Tensor Processing Unit: Architecture by technofiend on Tuesday March 26, 2024 @12:22AM (#39824246)
The Palm Pilot V had a dockable cell phone modem, but the connectivity wasn't integrated into the OS. It worked but only as a demonstration. Then Palm released a model with integrated data, but the BlackBerry came out the same year. You can be first and still if someone comes along with a much more compelling product, that's the end of you.

Google has a few years left as a search company, but their enshittification of results has doomed them to replacement by LLMs. They seem to have forgotten Google pushed out their predecessors by having the best search results. Targeted advertisements don't qualify.

Re:Google's First Tensor Processing Unit: Architecture by Rinzler89 on Tuesday March 26, 2024 @05:07AM (#39825646)
>and the batteries only lasted like four hours

Still more than the OG Steam Deck today :)

Re:Google's First Tensor Processing Unit: Architecture by sofixa on Tuesday March 26, 2024 @06:09AM (#39826012)
Vastly depends on the game played and the settings. In a plane (so airplane mode, with Bluetooth headset) I played Hitman Absolution for 3 hours and still had 50%+ of the battery left. It was on minimal brightness because it was dark and didn't need more, but still.
Re:Google's First Tensor Processing Unit: Architecture by Rinzler89 on Tuesday March 26, 2024 @10:05AM (#39827988)
Yeah, no need to take a (semi)joke literally and go all technical to debunk it. Though without optimizations, battery life on the deck was lucky to hit 2h at first before valve brought in updates and people learned they had to cap resolution and FPS to increase battery life.
Re:Google's First Tensor Processing Unit: Architecture by chillfox on Monday March 25, 2024 @10:32PM (#39823746)
Kodak
Re:Google's First Tensor Processing Unit: Architecture by BolexNOLA on Monday March 25, 2024 @10:58PM (#39823884)
Man I remember my last semester of college taking a history of photography course that was only offered every 3-4 years by a pretty legendary professor. The the day before the first day of class (or super close), Eastman Kodak declared bankruptcy after what? 110 years?

He scrapped his day 1 lecture and threw together a talk - with photos of course - about Kodak and how an intrepid engineer developed then the company foolishly hid the first digital camera because it would compete with their film line.

Incredible lecturer for sure haha

Re:Google's First Tensor Processing Unit: Architecture by vineyardmike on Monday March 25, 2024 @11:33PM (#39824042)
I think the TPU is simple. They do sell it (via cloud), but they focus on themselves first. When there was no shortage of compute, it was an also-ran in the ML hardware market. Now it’s trendy.

ChatGPT v Google is a far crazier history. Not only did Google invent Transformers, not only did Google open-source PaLM and Bert, but they even built chat tuned LLM chat bots and let employees talk with it. This isn’t a case where they were avoiding for disruption or protecting search - they genuinely didn’t see its potential. Worse, they got so much negative publicity over it that they considered it an AI safety issue to release. If that guy hadn’t gone to the press and claimed LaMDA was sentient than they may have entirely open sourced it like PaLM. This would likely mean that GPT-3 was open sourced and maybe never chat tuned either.

GPT-2 was freely available and OpenAI showed off GPT-3 freely as a parlor trick before ChatGPT came out. ChatGPT was originally the same - fun text generation as chat not a full product.

TLDR - Tensors probably didn’t have a lot of value until NVidia because scarce and they actively invented the original ChatGPT and “AI Safety” concerns caused them to lock it down.

Re:Google's First Tensor Processing Unit: Architecture by snats on Monday March 25, 2024 @11:13PM (#39823958)
to this day i am impressed that they have not figured out how to embed advertisments to bard outputs so that they can go free.
Re:Google's First Tensor Processing Unit: Architecture by romanovcode on Monday March 25, 2024 @10:57PM (#39823876)
Pretty sure it is because if ChatGPT likes would update as frequently as google website index it would render search engines like google obsolete and thus make their revenue nonexistent.
Re:Google's First Tensor Processing Unit: Architecture by willvarfar on Tuesday March 26, 2024 @03:34AM (#39825078)
OpenAI lured everyone away from Google with way higher pay.

https://www.linkedin.com/posts/eolver_googles-defense-agains...