News for hackers. Stuff that matters.
Re:Outperforming LLMs with less training data and smaller mo... by threeseed on Thursday September 21, 2023 @10:18PM (#37607114)
> For instance, serving a single 175 billion LLM requires at least 350GB of GPU memory using specialized infrastructure

Apple ships the Mac Studio which support up to 144GB of usable GPU memory.

Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space.

Re:Outperforming LLMs with less training data and smaller mo... by rubyn00bie on Thursday September 21, 2023 @10:49PM (#37607281)
Apple's hardware advantage is something I'm hoping to see really unleashed with the M3 generation. The fact the A17 Pro has ray tracing support is giving me hope they can catch up with the incumbents quickly. It's honestly the one thing that has kept me away from their newer hardware-- I primarily use my computer at a desk and PC hardware (GPU primarily) is leagues ahead of what their very best can do. I honestly can't justify spending nearly 4k when Linux works great for everything I do professionally AND when I'm done with work I can still play a game.
Re:Outperforming LLMs with less training data and smaller mo... by behnamoh on Thursday September 21, 2023 @11:09PM (#37607377)
3 most important things for Apple are:

- HW design

- UI

- UX

That Macs come with awesome VRAM is only because #3. Apple has no explicit incentive to make LLM-ready machines.

Re:Outperforming LLMs with less training data and smaller mo... by leetharris on Thursday September 21, 2023 @11:41PM (#37607576)
You can find many leaked email threads from Apple where they talk about the value of high powered local compute. It is a fundamental part of their strategy. They do not want the cloud approach for high end compute like Google, they want you to have high end and powerful devices in your hands.
Re:Outperforming LLMs with less training data and smaller mo... by flangola7 on Friday September 22, 2023 @12:03AM (#37607701)
Got a link to those emails?
Re:Outperforming LLMs with less training data and smaller mo... by rubyn00bie on Thursday September 21, 2023 @11:21PM (#37607441)
I disagree. ML is eating the world by storm and one thing Apple loves is high margins. Folks like myself would gladly pay that margin to someone besides Nvidia. I paid $2k for my last GPU and was happy to do it. That doesn’t even factor the fact that businesses would pay even more when the alternative are machines that start in the tens of thousands for just the GPU ability. GPUs right now are carrying fat margins and that makes more than enough incentive for Apple IMHO.
Re:Outperforming LLMs with less training data and smaller mo... by behnamoh on Thursday September 21, 2023 @11:35PM (#37607532)
But you wouldn't be able to upgrade an Apple Mac when the next generation of LLMs is out.
Re:Outperforming LLMs with less training data and smaller mo... by nborwankar on Friday September 22, 2023 @10:26AM (#37612509)
You do get some money back for a trade-in depending on how old the machine is.
Re:Outperforming LLMs with less training data and smaller mo... by stevenhuang on Thursday September 21, 2023 @11:12PM (#37607393)
That number is not quantized either. Quantize 175B param at 4 bits then it'd fit in ~120GB VRAM (one can fit 34B models at 4 bit quantization in a single RTX3090 24GB VRAM)
Re:Outperforming LLMs with less training data and smaller mo... by brucethemoose2 on Thursday September 21, 2023 @11:35PM (#37607535)
> Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space

Is there any framework that can batch LLMs on Metal? I don't think GGML or MLC have it yet.

Otherwise that is just another reason they wouldn't be good for LLM hosting at this moment.

Anyway, the real disruptor is Intel. They could theoretically barge in with a 2x48GB Arc card and undercut the market that AMD/Nvidia refuse to dive into because of their Pro card clients.

Re:Outperforming LLMs with less training data and smaller mo... by astrange on Friday September 22, 2023 @01:01AM (#37608071)
Re:Outperforming LLMs with less training data and smaller mo... by brucethemoose2 on Friday September 22, 2023 @01:51AM (#37608355)
Amazing!

Its really a shame that Apple (and AMD/Intel or pretty much any other infrence vendor) are not directly contributing to llama.cpp. The feature set is amazing and growing at a stunning pace.

Re:Outperforming LLMs with less training data and smaller mo... by nomel on Friday September 22, 2023 @12:47PM (#37614328)
I imagine the legal concerns around it are part of the reason. Where I am, our legal team has been very direct, that anything llama based cannot touch a work computer.
Re:Outperforming LLMs with less training data and smaller mo... by stringtoint on Friday September 22, 2023 @05:36PM (#37618101)
Curious to know why this is the case? I haven't played around with LLAMA yet but I figured it being open-source would make it more trustworthy than models provided by OpenAI.
Re:Outperforming LLMs with less training data and smaller mo... by nomel on Friday September 22, 2023 @06:02PM (#37618368)
> it being open-source

Absolutely not. The LLAMA license [1] is clear that it's not open source. It's for non-commercial, research only, and only by explicit permission from Meta. The weights were leaked, on 4chan [2], illegally, according to the license. Very very few people are using it legally. This interpretation is clear from its wording, and also matches the interpretation of our flock of lawyers.

[1] https://github.com/facebookresearch/llama/issues/266

[2] https://levelup.gitconnected.com/metas-chatgpt-is-now-illega...

Re:Outperforming LLMs with less training data and smaller mo... by astrange on Friday September 22, 2023 @07:00PM (#37618896)
Llama 2 doesn't use that license and llama.cpp isn't limited to using those models anyway.
Re:Outperforming LLMs with less training data and smaller mo... by nomel on Friday September 22, 2023 @07:33PM (#37619147)
True. That gives these companies a little less than two months to start contributing, with the easier license restrictions in mind.

But, Llama 2 is not open source [1].

[1] https://blog.opensource.org/metas-llama-2-license-is-not-ope...

Re:Outperforming LLMs with less training data and smaller mo... by brucethemoose2 on Saturday September 23, 2023 @12:53AM (#37620663)
I wonder what legal teams (such as y'alls legal dept) think of the new license.
Re:Outperforming LLMs with less training data and smaller mo... by stringtoint on Sunday September 24, 2023 @05:11PM (#37637232)
You're correct, I confused it with LLAMA being more business-friendly (at least in theory) than some other models out there.
Re:Outperforming LLMs with less training data and smaller mo... by oezi on Friday September 22, 2023 @04:24AM (#37609412)
I am wondering who will be the first to raise RAM capacity of their hardware offerings dramatically to win the LLM crowd. Seems like the way to win market share.
Re:Outperforming LLMs with less training data and smaller mo... by marcyb5st on Friday September 22, 2023 @06:25AM (#37610234)
Agreed. I would love to have something with the FLOPs of a 3070/4060 and 80 gigs of even slower VRAM (not necessarily HBM/GDDR6x) that can run the XXB models.