Doesn't OpenAI depend mostly on its relationship/partnership with Microsoft to get GPUs to inference on?
Thanks for the links, interesting book!
That is, as a research person using our GPUs and TPUs I see first hand how choices from the high level python level, through Jax, down to the TPU architecture all work together to make training and inference efficient. You can see a bit of that in the gif on the front page of the book. https://jax-ml.github.io/scaling-book/
I also see how sometimes bad choices by me can make things inefficient. Luckily for me if my code/models are running slow I can ping colleagues who are able to debug at both a depth and speed that is quite incredible.
And because were on HN I want to preemptively call out my positive bias for Google! It's a privilege to be able to see all this technology first hand, work with great people, and do my best to ship this at scale across the globe.
Even then, I think that their primary use case is going to be consumer grade good AI on phones. I dunno why Gemma QAT model fly so low on the radar, but you can basically get full scale Llamma 3 like performance from a single 3090 now, at home.
Google has already started the process of letting companies self-host Gemini, even on NVidia Blackwell GPUs.
Although imho, they really should bundle it with their TPUs as a turnkey solution for those clients who haven't invested in large scale infra like DCs yet.
And also, Google's track record with hardware.
My point was a bit more specific though. To elaborate, I know of a number of publicly traded companies (USD $200M+ market cap) globally which have identified use cases for onprem AI and want to implement them actively but cannot, because they lack the knowhow to work with onprem, and hiring talent to implement that is just extremely expensive. Google should simply provide it as a turnkey bundle and milk them for it.
They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.
But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them.
When I was at AWS ProServe, we didn’t even bother coming up with talking points when competing with GCP except to point out how they abandon services. Was it partially FUD? Probably. But it worked.
there are few groups as talented at losing a head start as google.
What are the chances of abandoning TPU-related projects where the company literally invested billions in infrastructure? Zero.
All things that Google is remarkably bad at.
Penny-wise pound-foolish.
https://tomforsyth1000.github.io/blog.wiki.html#%5B%5BWhy%20...
So it isn't like Google designed a TPU for a specific model or architecture. They're pretty general purpose in a narrow field (oxymoron, but you get the point).
The set of operations Google designed into a TPU is very similar to what nvidia did, and it's about as broadly capable. But Google owns the IP and doesn't pay the premium and gets to design for their own specific needs.
Personally right now I see one clear leader and one group going 0-99 like a five sigma cosmic ray: Anthropic and the PRC. But this is because I believe/know that all the benchmarks are gamed as hell, its like asking if a movie star had cosmetic surgery. On quality, Opus 4 is 15x the cost and sold out / backordered. Qwen 3 is arguably in next place.
In both of those cases, extreme quality expert labeling at scale (assisted by the tool) seems to be the secret sauce.
Which is how it would play out if history is any guide: when compute as a scaling lever starts to flatten, you expert label like its 1987 and claim its compute and algorithms until the government wises up and stops treating your success persobally as a national security priority. It's the easiest trillion Xi Xianping ever made: pretending to think LLMs are AGI too, fast following for pennies on the dollar, and propping up a stock market bubble to go with the fentanyl crisis? 9-D chess. It's what I would do about AI if I were China.
Time will tell.
All the LLM vendors are going to have to cope with the fact that they're lighting money on fire, and Google have the paying customers (advertisers) and with the user-specific context they get from their LLM products, one of the juciest and most targetable ad audiences of all time.
I would only offer one disagreement with your post: There will not be a single winner in LLMs. The landscape is so large that we will have multiple winners in different areas. Example: Google might fail in B2C LLM (chatbot that answers your questions), but will certainly be (wildly?) successful in B2B for adverts.
> llama.cpp an other inference engines auto add a <bos> - DO NOT add TWO <bos> tokens! You should ignore the <bos> when prompting the model!
That makes the want to try exactly that? Weird
And folks at LMSys: https://lmsys.org/blog/
Large Model Systems (LMSYS Corp.) is a 501(c)(3) non-profit focused on incubating open-source projects and research. Our mission is to make large AI models accessible to everyone by co-developing open models, datasets, systems, and evaluation tools. We conduct cutting-edge machine learning research, develop open-source software, train large language models for broad accessibility, and build distributed systems to optimize their training and inference.Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines.
I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu monsters. so long as your model fitted in (combined) vram, you could job was a goodun.
To scale the secret ingredient was industrial amounts of cash. Sure we had DGXs (fun fact, nvidia sent literal gold plated DGX machines) but they wernt dense, and were very expensive.
Most large companies have robust RPC, and orchestration, which means the hard part isn't routing the message, its making the model fit in the boxes you have. (thats not my area of expertise though)
I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!), you "only" pay for the request-specific raw compute and the memory storage+bandwidth for the activations. And the proprietary models are now huge, highly-quantized extreme-MoE models where the former factor (model size) is huge and the latter (request-specific compute) has been correspondingly minimized - and where it hasn't, you're definitely paying "pro" pricing for it. I think this goes a long way towards explaining how inference at scale can work better than locally.
(There are "tricks" you could do locally to try and compete with this setup, such as storing model parameters on disk and accessing them via mmap, at least when doing token gen on CPU. But of course you're paying for that with increased latency, which you may or may not be okay with in that context.)
Kind of unrelated, but this comment made me wonder when we will start seeing side channel attacks that force queries to leak into each other.
On the other hand, I've learned that when I ask questions about security to experts in a field (who are not experts in security) I almost always get convincing hand waves, and they are almost always proven to be completely wrong.
Sigh.
Quite the opposite. Context caching requires state (K/V cache) close to the VRAM. Streaming requires state. Constrained decoding (known as Structured Outputs) also requires state.
Unless something has dramatically changed, the model is stateless. The context cache needs to be injected before the new prompt, but for what I understand (and please do correct me if I'm wrong) the the context cache isn't that big, like in the order of a few tens of kilobytes. Plus the cache saves seconds of GPU time, so having an extra 100ms of latency is nothing compare to a cache miss. so a broad cache is much much better than a narrow local cache.
But! even if its larger, Your bottleneck isn't the network, its waiting on the GPUs to be free[1]. So whilst having the cache really close ie in the same rack, or same machine, will give the best performance, it will limit your scale (because the cache is only effective for a small number of users)
[1] a 100megs of data shared over the same datacentre network every 2-3 seconds per node isn't that much, especially if you have a partitioned network (ie like AWS where you have a block network and a "network" network)
"we do 1970s mainframe style timesharing"
there, that was easy
If the requests were regularly spaced, and they certainly won’t be, but for the sake of argument, then 1 machine could serve 17,000 requests per day, or 120,000 per week. At that rate, you’d need about 5,600 machines to serve 700M requests. That’s a lot to me, but not to someone who owns a data center.
Yes, those 700M users will issue more than 1 query per week and they won’t be evenly spaced. However, I’d bet most of those queries will take well under 1 second to answer, and I’d also bet each machine can handle more than one at a time.
It’s a large problem, to be sure, but that seems tractable.
Here's an entry-level link I found quickly on Google, OP: https://medium.com/@wearegap/a-brief-introduction-to-optimiz...
Not sure if you were just joking or really believe that, but for other peoples’ sake, it’s wildly wrong.
I mean i get the idea but sounds so incredibly rare it would mean absolutely nothing optimisation wise.
It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.
> It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.
They're definitely running cluster knoppix.
:-)
I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem that a) needs to be solved from a business point of view, and b) should be solved from a civic-minded point of view.
So not unlike an LLM then?
No solution is proposed here but the truth is on its face comic.
> The only solution to those people starving to death is to kill the people that benefit from them starving to death.
There are solutions other than "to kill the people that benefit", such as what have existed for many years, including but not limited to:
- Efforts such as the recently emasculated USAID[0].
- Humanitarian NGO's[1] such as the World Central Kitchen[2]
and the Red Cross[3].
- The will of those who could help to help those in need[4].
0 - https://en.wikipedia.org/wiki/United_States_Agency_for_Inter...
1 - https://en.wikipedia.org/wiki/Non-governmental_organization
2 - https://wck.org/
3 - https://en.wikipedia.org/wiki/International_Red_Cross_and_Re...
This is a political will, empathy, and leadership problem. Not an engineering problem.
>> This is a political will, empathy, and leadership problem. Not an engineering problem.
> Those problems might be more tractable if all of our best and brightest were working on them.
The ability to produce enough food for those in need already exists, so that problem is theoretically solved. Granted, logistics engineering[0] is a real thing and would benefit from "our best and brightest."
What is lacking most recently, based on empirical observation, is a commitment to benefiting those in need without expectation of remuneration. Or, in other words, empathetic acts of kindness.
Which is a "people problem" (a.k.a. the trio I previously identified).
This sort of "there are bad things in the world, therefore focusing on anything else is bad" thinking is generally misguided.
> Engineers have basically nothing to do with either of those.
The logic here is “If A is actively working to develop capabilities for B, which B offers up to C who then uses it to do D, then A cannot claim to have nothing to do with D.”
And if AI was truly the holy grail that it's being sold as then there wouldn't be 700M users per week wasting all of these resources as heavily as we are because generative AI would have already solved for something better. It really does seem like these platforms are, and won't be, anywhere as useful as they're continuously claimed to be.
Just like Tesla FSD, we keep hearing about a "breakaway" model and the broken record of AGI. Instead of getting anything exceptionally better we seem to be getting models tuned for benchmarks and only marginal improvements.
I really try to limit what I'm using an LLM for these days. And not simply because of the resource pigs they are, but because it's also often a time sink. I spent an hour today testing out GPT-5 and asking it about a specific problem I was solving for using only 2 well documented technologies. After that hour it had hallucinated about a half dozen assumptions that were completely incorrect. One so obvious that I couldn't understand how it had gotten it so wrong. This particular technology, by default, consumes raw SSE. But GPT-5, even after telling it that it was wrong, continued to give me examples that were in a lot of ways worse and kept resorting to telling me to validate my server responses were JSON formatted in a particularly odd way.
Instead of continuing to waste my time correcting the model I just went back to reading the docs and GitHub issues to figure out the problem I was solving for. And that led me down a dark chain of thought: so what happens when the "teaching" mode rethinks history, or math fundamentals?
I'm sure a lot of people think ChatGPT is incredibly useful. And a lot of people are bought into not wanting to miss the boat, especially those who don't have any clue to how it works and what it takes to execute any given prompt. I actually think LLMs have a trajectory that will be similar to social media. The curve is different and I, hopefully, don't think we've seen the most useful aspects of it come to fruition as of yet. But I do think that if OpenAI is serving 700M users per week then, once again, we are the product. Because if AI could actually displace workers en masse today you wouldn't have access to it for $20/month. And they wouldn't offer it to you at 50% off for the next 3 months when you go to hit the cancel button. In fact, if it could do most of the things executives are claiming then you wouldn't have access to it at all. But, again, the users are the product - in very much the same way social media played into.
Finally, I'd surmise that of those 700M weekly users less than 10% of those sessions are being used for anything productive that you've mentioned and I'd place a high wager that the 10% is wildly conservative. I could be wrong, but again - we'd know about that if it were the actual truth.
Is everything you spend resources on truly productive?
Who determines whether something is worth it? Is price/willingness of both parties to transact not an important factor?
I don't think ChatGPT can do most things I do. But it does eliminate drudgery.
We know a few things: LLMs are not efficient, LLMs are consuming more water than traditional compute, we know the providers know but they haven't shared any tangible metrics, and the build process involves, also, an exceptional amount of time, wattage and water.
For me it's: if you have access to a supercomputer do you use it to tell you a joke or work on a life saving medicine?
We didn't have these tools 5 years ago. 5 years ago you dealt with said "drudgery". On the other hand you then say it can't do "most things I do". It seems as though the lines of fatalism and paradox are in full force for a lot of the arguments around AI.
I think the real kicker for me this week (and it changes week-over-week, which is at least entertaining) is when Paul Graham told his Twitter feed [1] a "hotshot" programmer is writing 10k LOC that are not "bug-filled crap" in 12 hours. That's 14 LOC per minute. Compared to industry norms of 50-150 LOC per 8 hour day. Apparently,this "hot-shot" is not "naive", though, implying that it's most definitely legit.
[0] https://www.sciencenews.org/article/ai-energy-carbon-emissio... [1] https://x.com/paulg/status/1953289830982664236
The stuff I'm punting isn't stuff I can automate. It's stuff like, "build me a quick command line tool to model passes from this set of possible orbits" or "convert this bulleted list to a course articulation in the format preferred by the University of California" or "Tell me the 5 worst sentences in this draft and give me proposed fixes."
Human assistants that I would punt this stuff to also consume a lot of wattage and power. ;)
> We didn't have these tools 5 years ago. 5 years ago you dealt with said "drudgery". On the other hand you then say it can't do "most things I do".
I'm not sure why you think this is paradoxical.
I probably eliminate 20-30% of tasks at this point with AI. Honestly, it probably does these tasks better than I would (not better than I could, but you can't give maximum effort on everything). As a result, I get 30-40% more done, and a bigger proportion of it is higher value work.
And, AI sometimes helps me with stuff that I -can't- do, like making a good illustration of something. It doesn't surpass top humans at this stuff, but it surpasses me and probably even where I can get to with reasonable effort.
I am not an anti-LLM’er here but having models that are this power hungry and this generalisable makes no sense economically in the long term. Why would the model that you use to build a command tool have to be able to produce poetry? You’re paying a premium for seldom used flexibility.
Either the power drain will have to come down, prices at the consumer margin significantly up or the whole thing comes crashing down like a house of cards.
A human eats 2000 kilocalories of food per day.
Thus, sitting around for an hour to do a task takes 350kJ of food energy. Depending on what people eat, it's 350kJ to 7000kJ of fossil fuel energy in to get that much food energy. In the West, we eat a lot of meat, so expect the high end of this range.
The low end-- 350kJ-- is enough to answer 100-200 ChatGPT requests. It's generous, too, because humans also have an amortized share of sleep and non-working time, other energy inputs/uses to keep them alive, eat fancier food, use energy for recreation, drive to work, etc.
Shoot, just lighting their part of the room they sit in is probably 90kJ.
> I am not an anti-LLM’er here but having models that are this power hungry and this generalisable makes no sense economically in the long term. Why would the model that you use to build a command tool have to be able to produce poetry? You’re paying a premium for seldom used flexibility.
Modern Mixture-of-Experts (MoE) models don't activate the parameters/do the math related to poetry, but just light up a portion of the model that the router expects to be most useful.
Of course, we've found that broader training for LLMs increases their usefulness even on loosely related tasks.
> Either the power drain will have to come down, prices at the consumer margin significantly up
I think we all expect some mixture of these: LLM usefulness goes up, LLM cost goes up, LLM efficiency goes up.
I feel like if you're going to go down the route of the energy consumption needed to sustain the entire human organism, you have to do that on the other side as well - as the actual activation cost of human neurons and articulating fingers to operate a keyboard won't be in that range - but you went for the low ball so I'm not going to argue that, as you didn't argue some of the other stuff that sustains humans.
But I will argue the wider implication of your comment that a like-for-like comparison is easy - it's not, so leaving it in the neuron activation space energy cost would probably be simpler to calculate, and there you'd arrive at a smaller ChatGPT ratio. More like 10-20, as opposed to 100-200. I will concede to you that economies of scale mean that there's an energy efficiency in sustaining a ChatGPT workforce compared to a human workforce, if we really want to go full dystopian, but that there's also outsized energy inefficiency in needing the industry and using the materials to construct a ChatGPT workforce large enough to sustain the economies of scale, compared to humans which we kind of have and are stuck with.
There is a wider point that ChatGPT is less autonomous than an assistant, as no matter the tenure with it, you'll not give it the level of autonomy that a human assistant would have as it would self correct to a level where you'd be comfortable with that. So you need a human at the wheel, which will spend some of that human brain power and finger articulation, so you have to add that to the scale of the ChatGPT workflow energy cost.
Having said all that - you make a good point with MoE - but the router activation is inefficient; and the experts are still outsized to the processing required to do the task at hand - but what I argue is that this will get better with further distillation, specialisation and better routing however only for economically viable task pathways. I think we agree on this, reading between the lines.
I would argue though (but this is an assumption, I haven't seen data on neuron activation at task level) that for writing a command-line tool, the neurons still have to activate in a sufficiently large manner to parse a natural language input, abstract it and construct formal language output that will pass the parsers. So you would be spending a higher range of energy than for an average Chat GPT task
In the end - you seem to agree with me that the current unit economics are unsustainable, and we'll need three processes to make them sustainable - cost going up, efficiency going up and usefulness going up. Unless usefulness goes up radically (which it won't due to scaling limitations of LLM's), full autonomy won't be possible, so the value of the additional labour will need to be very marginal to a human, which - given the scaling laws of GPU's - doesn't seem likely.
Meanwhile - we're telling the masses at large to get on with the programme, without considering that maybe for some classes of tasks it just won't be economically viable; which creates lock in and might be difficult disentangle in the future.
All because we must maintain the vibes that this technology is more powerful than it actually is. And that frustrates me, because there's plenty pathways where it's obvious it will be viable, and instead of doubling down on those, we insist on generalisability.
IDK. I didn't give human entry level employees that much autonomy. ChatGPT runs off and does things for a minute or two consuming thousands and thousands of tokens, which is a lot like letting someone junior spin for several hours.
Indeed, the cost is so low -- better to let it "see its vision through" than to interrupt it. A lot of the reason why I'd manage junior employees closely are to A) contain costs, and B) prevent discouragement. Neither of those apply here.
(And, you know -- getting the thing back while I remember exactly what I asked and still have some context to rapidly interpret the result-- this is qualitatively different from getting back work from a junior employee hours later).
> that maybe for some classes of tasks it just won't be economically viable;
Running an LLM is expensive. But it's expensive in the sense "serving a human costs about the same as a long distance phone call in the 90's." And the vast majority of businesses did not worry about what they were expending on long distance too much.
And the cost can be expected to decrease, even though the price will go up from "free." I don't expect it will go up too high; some players will have advantages from scale and special sauce to make things more efficient, but it's looking like the barriers to entry are not that substantial.
Open AI reportedly had a loss of $5B last year. That's really small for a service with hundreds of millions of users (most of which are free and not monetized in any way). That means Open AI could easily turn a profit with ads, however they may choose to implement it.
The person attempting to learn either (hopefully) figures out the AI model was wrong, or sadly learns the wrong material. The level of impact is probably quite relative to how useful the knowledge is one's life.
The good or bad news, depending on how you look at it, is that humans are already great at rewriting history and believing wrong facts, so I am not entirely sure an LLM can do that much worse.
Maybe ChatGPT might just kill of the ignorant like it already has? GPT already told a user to combine bleach and vinegar, which produces chlorine gas. [1]
Very, very few problems _need_ to be solved. Feeding yourself is a problem that needs to be solved in order for you to continue living. People solve problems for different reasons. If you don't think LLMs are valuable, you can just say that.
1. How to identify humanity's needs on all levels, including cosmic ones...(we're in the Space Age so we need to prepare ourselves for meeting beings from other places)
2. How to meet all of humanity's needs
Pointing this out regularly is probably necessary because the issue isn't why people are choosing what they're doing...it's that our systems actively disincentivize collectibely addressing these two problems in a way that doesn't sacrifice people's wellbeing/lives... and most people don't even think about it like this.
One of my colleagues was only 25, really smart in his field and became a professor less than 10 years later. But he was incredibly naive in everyday chores. Buying groceries or filing taxes resulted in major screw-ups regularly
The real answer is likely internal company politics and priorities. Google certainly has people with the technical skills to solve it but do they care and if they care can they allocate those skilled people to the task?
It’s quite intimidating how fast they can break down difficult concepts into first principles. I’ve witnessed this first hand and it’s beyond intimidating. Makes you wondering what you’re doing at this company… That being said, the caliber of folks I’m talking about is quite rare, like top 10% of top 1% teams at Google.
I don't like how the grand parent mystifies this. This problem is just normal engineering. Any good engineer could learn how to do it.
The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self estimates.
This stuff is well understood in public, and where a big name has something highly custom going on? Often as not it's a liability around attachment to some legacy thing. You run this stuff at scale by having the correct institutions and processes in place that it takes to run any big non-trivial system: that's everything from procurement and SRE training to the RTL on the new TPU, and all of the stuff is interesting, but if anyone was 10x out in front of everyone else? You'd be able to tell.
Signed, Someone Who Also Did Megascale Inference for a TOP-5 For a Decade.
Sounds analogous to the 60's and 70's i.e "even small programs run so close to hardware limits". If optimization and efficiency is dead in software engineering, it's certainly alive and well in LLM development.
However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast.
https://jax-ml.github.io/scaling-book/
In particular your questions are around inference which is the focus of this chapter https://jax-ml.github.io/scaling-book/inference/