We won't be running on GPUs forever. The innovation in the chip space will continue.
With as much margin as cloud takes, there will be increased interest in on-prem and mixed environments.
We run our own GPUs in production as well as having a cloud footprint.
Sure, but I'm seeing a divergence. There are a group of chips that are getting smaller and require less power and cooling that can do a lot of workloads that you used to need huge chips for, but there are the most advanced chips for the most advanced workloads getting bigger and more power hungry.
There will always be the big customers who need the big chips, and that is what will be nearly impossible for them to self host.
We are reaching the power:computational limits of silicon.
Both GPU and CPU.
“Moore’s Law’s dead,” Huang said, referring to the standard that the number of transistors on a chip doubles every two years. “And the ability for Moore’s Law to deliver twice the performance at the same cost, or at the same performance, half the cost, every year and a half, is over. It’s completely over, and so the idea that a chip is going to go down in cost over time, unfortunately, is a story of the past. Computing is a not a chip problem, it’s a software and chip problem,” Huang said.
What we are seeing now are software engineers offloading computationally expensive workloads to AIA's, more and more. This is enabled through the use of libraries like PyTorch and access to HPC levels of compute that were not broadly accessible before.
e.g. A million chickens could have a thousand times more muscle mass than two strong oxen, but I doubt anyone could plow a field even 2x faster with a million chickens.
So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently.
Hasn't that pretty much always been the case?
One thing we are seeing more and more of is composable fabrics where the PCIe bus is effectively being extended outside the case such that you log into a single instance, and instead of seeing just 8 AIA's, you now see 32+. This makes the coordination a lot easier.
I was talking about that, the fabric stuff is a non-sequitor, random hardware, doesn't make GPUs 2x in speed every 18 months, much less lead to a shift in everyday computing loads to GPUs.
1. Moore's Law also applies to GPUs.
2. If we could make use of 1000 cores for anything but long tail tasks, ye average CPU would have a lot more than 8 cores by now.
That's what the million chickens thing is about.
You can give me ultra-unobtianium fabric for "AIAs", it doesn't matter unless I have a algorithm that's massively parallelizable.
To be clear, I was responding to this from the OP above:
"and the coordination overhead increases exponentially too"
Cross node coordination is complicated and more easily managed with composable fabrics, which is why I brought it up.
--
I went back and read a bunch of your older comments here. At first, I thought it was personal but the tone that you use with me is carried across many comments that you've made to others. These also tend to be heavily downvoted.
I went and looked you up a bit more, it seems you had an exit to Google and then worked there for a number of years. There is something about how you're communicating that seems brilliant, but at the same time, is missing some level of EQ.
Like you're super smart, but also have some sort of ego about it all. As if you feel a deep need to somehow prove yourself as being "better" than the people around you. Saying things like this: "as if you are encountering them the first time in the thread"... is honestly, just rude.
My goal here is to help educate and discuss things as best as I know how. If that isn't good enough for you, honestly, just keep it to yourself. Unlike you, I'm not putting my ego into it. If you have a personal problem with me or my background or what I'm saying, my suggestion would be to stick with the technicals and leave the ego at the door.
Feel free to say whatever you want, but I won't respond to you any further.
There's a few things you got very wrong. I'm happy to talk about my background, but...I have a weird feeling you aren't actually interested.
Seems you thought I was putting you down personally so you needed to find out who I was. I don't know you from an 8 year old. You're brilliant too or whatever you need to hear.
Whole thing made me cringe. It's not worth it my friend. Makes you feel good until 30 seconds after you click post.
Going off on a long personal attack, claiming you did your OSINT research on your interlocutor, and they're victimizing you with their vicious claims that GPUs are limited by Moore's Law...not a good look.
Apology accepted.
Update: Weird, now you keep going after I accepted your apology. Now you're editing your comments to change things around to somehow make yourself look better. ¯\_(ツ)_/¯
GPU is a poor fit for what an MI300x is, given that it doesn't really have anything to do with graphics any more. The boards don't even have display outputs.
CDNA3 vs. RDNA3
I actually see a lot of parallels to bitcoin mining. It started on cpus, then gpus, then fpgas and soon after asics. We are starting to see something similar in the AI space with groq and tenstorrent.
The problem with asic's today is that in order to get to the same compute density of a AIA, due to AI's need for memory, you need significantly more hardware.
Right now, MI300x is the leader with 192gb, if you do the math on groq, it just doesn't work out. This is part of the reason why the stopped selling to retail and are just focused on hosting services.
It will be a while before the smaller chipmakers can get their hands on enough hbm to make a dent in the AIA market (due to AMD/NV hogging it all). Don't forget that AMD/NV can and will just come out with better designs too.
Or cheaper clouds...
I wonder how much lock-in the current clouds actually have. And if they'll fully take customers down or entire industries hostage like the mainframes did.
https://lambdalabs.com/blog/voltron-data-case-study-why-ml-t...
"While most other cloud companies offer contracts starting at one year, you can reserve a Lambda Reserved Cloud Cluster starting at six months."
If you're talking in-building flooding, not much going on there other than the racks themselves keep the equipment off the floor. All power is ran from overhead. A lot of modern "rack groups" (OCS, TSCIF, etc) have water limited water protection in mind.
Not to say we're free of challenges... but hey. We can make this happen
I have a couple racks in the space where car doors are expected, yet can still park. Think of the business continuity wins!
With 1Gb+ residential internet speeds, you could probably build a fairly robust and scalable platform like this.
How many people do you think have symmetric gigabit? The company is going to have to pay for employees to upgrade. And do some separation to keep from saturating link from either work or home.
Then there is latency. Most apps talk to a data store, which is now separated from clients. The load balancers are going to be separated from the backend. Also, database servers tend to be beefy servers so either need to shared it or someone hosts a rack.
Another is that need write software to really be distributed. There is difference between distributed when things break and distributed all the time over the internet.
The "build it and ship it to the customer" business is waning. The reason is because even the most deep pocketed customers no longer have data centers that can support the power and cooling needs of current GPUs. The latest from NVIDIA requires liquid cooling, tons of power, and solid floors (the racks are too heavy for raised floors).
Being able to run a modern server is becoming a very niche skill, one that most companies don't want or need to invest in.