News for hackers. Stuff that matters.
Re:Powering cost-efficient AI inference at scale with Cloud ... by deeviant on Tuesday November 28, 2023 @06:12PM (#38452910)
Google, structurally, sucks at anything but sucking down Ad money.

A good LLM allows one to basically bypass the internet for many use cases and drastically reduces ad time. As cool as LLM capabilities are the thing that delights me most about them is I can get high quality (YMMV) answers from the internet without all the godforsaken ad bullshit and wonky websites made only as mazes to keep users in ad gardens.

Google is stuck in one hell of an innovators dilemma, one that I can't really see them getting unstuck from.

As far as this particular product, it's touting cost efficiency, but that's not really dominating many decisions yet. There is an all-in sprint to get models trained and take their piece of the pie before the moats get too deep and AI hardware everywhere is back-ordered 12+ months.

Re:Powering cost-efficient AI inference at scale with Cloud ... by oezi on Tuesday November 28, 2023 @06:32PM (#38453152)
> good LLM allows one to basically bypass the internet for many use cases and drastically reduces ad time

Until they advertise during your Chat with the LLM. The question is just when this happens not if.

Re:Powering cost-efficient AI inference at scale with Cloud ... by fatherzine on Tuesday November 28, 2023 @07:22PM (#38453664)
If only. How about product placement seamlessly melded into any digital interaction? BRB, prompting ChatGPT for a YCombinator funding pitch.
Re:Powering cost-efficient AI inference at scale with Cloud ... by deeviant on Tuesday November 28, 2023 @07:10PM (#38453546)
Too true.

But in the beginning at least, there will be high competition which will hold down the ads a bit, with enshitifcation happening after the market entrenches around the winners.

Re:Powering cost-efficient AI inference at scale with Cloud ... by NBJack on Tuesday November 28, 2023 @09:40PM (#38454788)
LLMs are great at historical data. They however still suck at being kept up to date. I would expect that someone (if not already has) will find a cost efficient way to continuously index new data, but it would be interesting to see how it stacks up with the computing power needed for modern search index pipelines.
Re:Powering cost-efficient AI inference at scale with Cloud ... by deeviant on Wednesday November 29, 2023 @11:09AM (#38461204)
Yeah, when I said "many use cases", that mostly means, "use cases that don't require near-term knowledge." Which are the bulk of my internet searches.

But even know, ChatGPT 4 can use search to get update-to-date info and use it's knowledge store to augment the current information.

Re:Powering cost-efficient AI inference at scale with Cloud ... by humble_nimble on Thursday November 30, 2023 @12:35PM (#38476325)
"As far as this particular product, it's touting cost efficiency, but that's not really dominating many decisions yet"

Why do these LLM startups have such large investments and such large spending? It's because the infrastructure costs of a GenAI startup is huge. That's most of the money that's changing hands. So, from the VCs down to accelerator platforms cost efficiency is a huge decision factor on all of this.