Re:Running a 180B parameter LLM on a single Apple M2 Ultra
by AnthonyMouse on Thursday September 07, 2023 @06:36PM
(#37426572)
> Here’s another way to think about it. Why does ISA matter in CPUs?
Honestly the answer is that it mostly doesn't.
An ISA isn't viable without tooling, but that's why it's the first thing they all get. The only ISA with any significant moat is x86, and that's because there is so much legacy closed source software for it that people still need but would have to be emulated on any other architecture. And even that only works as long as x86 processors are competitive; if they fell behind then customers would just eat the emulation overhead on something else.
Other ISAs don't even have that. Would anybody actually be surprised if RISC-V took a huge chunk of out ARM's market share in the not too distant future?
Re:Running a 180B parameter LLM on a single Apple M2 Ultra
by foobiekr on Friday September 08, 2023 @12:44AM
(#37429430)
That's literally my point. The problem is that there's a massive amount of hidden infrastructure behind those that you don't see and that "oh look everyone has a big model" isn't as impressive as it sounds.
Re:Running a 180B parameter LLM on a single Apple M2 Ultra
by AnthonyMouse on Friday September 08, 2023 @03:52AM
(#37430771)
But the open source infrastructure is getting built too. And the infrastructure is mostly independent of the model. This is Falcon 180B running using the code from llama.cpp.
Here’s another way to think about it. Why does ISA matter in CPUs? There are minor issues around efficiencies of various kinds, but the real advantage of any mainstream ISA is, in part, the availability of tooling (hence this was a correct and heavy early focus for the RISCV effort) but also a lot of ecosystem things you don’t see: for example, Intel and Arm have truly mammoth test and verification suites that represent thousands++ of man years of investment.
OpenAI almost certainly has a massive invisible accumulated value at this point.
The actual models themselves are the output in the same way that a packaged CPU is the output. How you got there matters almost as much or more.