News for hackers. Stuff that matters.
Re:Outperforming LLMs with less training data and smaller mo... by brucethemoose2 on Thursday September 21, 2023 @11:35PM (#37607535)
> Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space

Is there any framework that can batch LLMs on Metal? I don't think GGML or MLC have it yet.

Otherwise that is just another reason they wouldn't be good for LLM hosting at this moment.

Anyway, the real disruptor is Intel. They could theoretically barge in with a 2x48GB Arc card and undercut the market that AMD/Nvidia refuse to dive into because of their Pro card clients.

Re:Outperforming LLMs with less training data and smaller mo... by astrange on Friday September 22, 2023 @01:01AM (#37608071)
Re:Outperforming LLMs with less training data and smaller mo... by brucethemoose2 on Friday September 22, 2023 @01:51AM (#37608355)
Amazing!

Its really a shame that Apple (and AMD/Intel or pretty much any other infrence vendor) are not directly contributing to llama.cpp. The feature set is amazing and growing at a stunning pace.

Re:Outperforming LLMs with less training data and smaller mo... by nomel on Friday September 22, 2023 @12:47PM (#37614328)
I imagine the legal concerns around it are part of the reason. Where I am, our legal team has been very direct, that anything llama based cannot touch a work computer.
Re:Outperforming LLMs with less training data and smaller mo... by stringtoint on Friday September 22, 2023 @05:36PM (#37618101)
Curious to know why this is the case? I haven't played around with LLAMA yet but I figured it being open-source would make it more trustworthy than models provided by OpenAI.
Re:Outperforming LLMs with less training data and smaller mo... by nomel on Friday September 22, 2023 @06:02PM (#37618368)
> it being open-source

Absolutely not. The LLAMA license [1] is clear that it's not open source. It's for non-commercial, research only, and only by explicit permission from Meta. The weights were leaked, on 4chan [2], illegally, according to the license. Very very few people are using it legally. This interpretation is clear from its wording, and also matches the interpretation of our flock of lawyers.

[1] https://github.com/facebookresearch/llama/issues/266

[2] https://levelup.gitconnected.com/metas-chatgpt-is-now-illega...

Re:Outperforming LLMs with less training data and smaller mo... by astrange on Friday September 22, 2023 @07:00PM (#37618896)
Llama 2 doesn't use that license and llama.cpp isn't limited to using those models anyway.
Re:Outperforming LLMs with less training data and smaller mo... by nomel on Friday September 22, 2023 @07:33PM (#37619147)
True. That gives these companies a little less than two months to start contributing, with the easier license restrictions in mind.

But, Llama 2 is not open source [1].

[1] https://blog.opensource.org/metas-llama-2-license-is-not-ope...

Re:Outperforming LLMs with less training data and smaller mo... by brucethemoose2 on Saturday September 23, 2023 @12:53AM (#37620663)
I wonder what legal teams (such as y'alls legal dept) think of the new license.
Re:Outperforming LLMs with less training data and smaller mo... by stringtoint on Sunday September 24, 2023 @05:11PM (#37637232)
You're correct, I confused it with LLAMA being more business-friendly (at least in theory) than some other models out there.