News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by detrites on Friday March 31, 2023 @04:58PM (#35393558)
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

Ok I answered my own question.

Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Friday March 31, 2023 @05:28PM (#35393921)
>What has everyone been doing wrong all these years

So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost that the business either needs to recuperate or turn into a competitive advantage. So everyone who scales up to this point doesn't release model weights.

The model in question - LLaMA - isn't even a public model. It leaked and people copied[0] it. But because such a large model leaked, now people can actually work on iterative improvements again.

Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers. Contributions-in-kind through distributed computing (e.g. a "GPT@home" project) would require significant changes to training methodology[1]. Further compounding this, the state-of-the-art is actually kind of a trade secret now. Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication.

[0] I'm avoiding the use of the verb "stole" here, not just because I support filesharing, but because copyright law likely does not protect AI model weights alone.

[1] AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.

Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @05:35PM (#35393979)
AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.

You can't if you have one 12gb gpu. You can if you have couple of dozens. And then petals-style training is possible. It is all very very new and there are many unsolved hurdles, but I think it can be done.

Re:Llama.cpp 30B runs with only 6GB of RAM now by dplavery92 on Friday March 31, 2023 @06:09PM (#35394356)
Sure, but when one 12gb GPU costs ~$800 new (e.g. for the 3080 LHR), "a couple of dozens" of them is a big barrier to entry to the hobbyist, student, or freelancer. And cloud computing offers an alternative route, but, as stated, distribution introduces a new engineering task, and the month-to-month bills for the compute nodes you are using can still add up surprisingly quickly.
Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @06:23PM (#35394546)
We are talking groups, not individuals. I think it is quite possible for couple of hundreds of people to cooperate and train something at least as big as LLaMa 7B in a week or two.
Re:Llama.cpp 30B runs with only 6GB of RAM now by webnrrd2k on Friday March 31, 2023 @06:27PM (#35394585)
Maybe a good candidate for the SETI@home treatment?
Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @06:31PM (#35394635)
It is a good candidate. Tech is good 6-18 months away, though.
Re:Llama.cpp 30B runs with only 6GB of RAM now by nullsense on Friday March 31, 2023 @07:35PM (#35395229)
How much faster can we develop the tech if we leverage GPT-4 to do it?
Re:Llama.cpp 30B runs with only 6GB of RAM now by 3np on Friday March 31, 2023 @08:47PM (#35395800)
One thing I don't understand: If it's possible to chunk and parallelize it, is it not relatively straightforward to do these chunks sequentially on a single GPU with a roughly linear increase in runtime? Or are the parallelized computations actually interdependent and involving message-passing, making this unfeasible?
Re:Llama.cpp 30B runs with only 6GB of RAM now by latency-guy2 on Friday March 31, 2023 @10:57PM (#35396618)
Data moving back and forth from CPU to to bus to GPU and back again for however many chunked model parts you have would increase training time far beyond what you would be willing to invest, not to mention how inefficient and power intensive it is, far more power needed than doing just CPU only or GPU only training. Back to the time part - it's not linear at all. IMO its easily quadratic.

It's not unfeasible, in fact that's how things were done before lots of improvements to the various libraries in essence, many corps still have poorly built pipelines that spend a lot of time in CPU land and not enough in GPU land.

Just an FYI as well - intermediate outputs of models are used in quite a bit of ML, you may see them in some form being used for hyperparameter optimization and searching.

Re:Llama.cpp 30B runs with only 6GB of RAM now by seydor on Friday March 31, 2023 @06:18PM (#35394466)
> we don't really have a way for the FOSS community to pool together that much money

There must be open source projects with enough money to pool into such a project. I wonder whether wikimedia or apache are considering anything.

Re:Llama.cpp 30B runs with only 6GB of RAM now by sceadu on Friday March 31, 2023 @08:06PM (#35395484)
Maybe we can repurpose the SETI@home infrastructure :)
Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Friday March 31, 2023 @11:26PM (#35396801)
BOINC might be usable but the existing distributed training setups assume all nodes have very high speed I/O so they can trade gradients and model updates around quickly. The kind of setup that's feasible for BOINC is "here's a dataset shard, here's the last epoch, send me back gradients and I'll average them with the other ones I get to make the next epoch". This is quite a bit different from, say, the single-node case which is entirely serial and model updates happen every step rather than epoch.
Re:Llama.cpp 30B runs with only 6GB of RAM now by totony on Saturday April 01, 2023 @02:03AM (#35397575)
Or big cloud platform could give some compute for free, give back some of the profit they get from oss.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Tryk on Friday March 31, 2023 @08:21PM (#35395609)
>Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers.

How so? Why couldn't we just start a gofundme/kickstarter to fund the training of an open-source model?

Re:Llama.cpp 30B runs with only 6GB of RAM now by hackernewds on Friday March 31, 2023 @09:43PM (#35396185)
who will be entrusted to benefit from this? recall that OpenAI did begin as an open source project. and that they chose to go the capitalist route, despite initially explicitly stating on their site they were a non-profit
Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Friday March 31, 2023 @11:59PM (#35396963)
OpenAI specifically cited scaling costs as a reason for why they switched their org structure from non-profit to "capped profit"[0].

You could potentially crowdfund this, though I should point out that this was already tried and Kickstarter shut it down. The effort in question, "Unstable Diffusion", was kinda sketchy, promising a model specifically tuned for NSFW work. What you'd want is an organization that's responsible, knows how to use state of the art model architectures, and at least is willing to try and stop generative porn.

Which just so happens to be Stability AI. Except they're funded as a for-profit on venture capital, not as a something you can donate to on Kickstarter or Patreon.

If they were to switch from investor subsidy to crowdfunding, however, I'm not entirely sure people would actually be lining up to bear the costs of training. To find out why we need to talk about motive. We can broadly subdivide the users of generative AI into a few categories:

- Companies, who view AI as a way to either juice stock prices by promising a permanent capitalist revolution that will abolish the creative working class. They do not care about ownership, they care about balancing profit and loss. Insamuch as they want AI models not controlled by OpenAI, it is a strategic play, not a moral one.

- Artists of varying degrees of competence who use generative AI to skip past creative busywork such as assembling references or to hack out something quickly. Insamuch as they have critiques of how AI is owned, it is specifically that they do not want to be abolished by capitalists using their own labor as ground meat for the linear algebra data blender. So they are unlikely to crowdfund the thing they are angry is going to put them out of a job.

- No-hopers and other creatively bankrupt individuals who have been sold a promise that AI is going to fix their lack of talent by making talent obsolete. This is, of course, a lie[2]. They absolutely would prefer a model unencumbered by filters on cloud servers or morality clauses in licensing agreements, but they do not have the capital in aggregate to fund such an endeavor.

- Free Software types that hate OpenAI's about-face on open AI. Oddly enough, they also have the same hangups artists do, because much of FOSS is based on copyleft/Share-Alike clauses in the GPL, which things like GitHub Copilot is not equipped to handle. On the other hand they probably would be OK with it if the model was trained on permissive sources and had some kind of regurgitation detector. Consider this one a wildcard.

- Evildoers. This could be people who want a cheaper version of GPT-4 that hasn't been Asimov'd by OpenAI so they can generate shittons of spam. Or people who want a Stable Diffusion model that's really good at making nonconsensual deepfake pornography so they can fuck with people's heads. This was the explicit demographic that "Unstable Diffusion" was trying to target. Problem is, cybercriminals tend to be fairly unsophisticated, because the people who actually know how to crime with impunity would rather make more money in legitimate business instead.

Out of five demographics I'm aware of, two have capital but no motive, two have motive but no capital, and one would have both - but they already have a sour taste in their mouth from the creep-tech vibes that AI gives off.

[0] In practice the only way that profit cap is being hit is if they upend the economy so much that it completely decimates all human labor, in which case they can just overthrow the government and start sending out Terminators to kill the working class[1].

[1] God damn it why do all the best novel ideas have to come by when I'm halfway through another fucking rewrite of my current one

[2] Getting generative AI to spit out good writing or art requires careful knowledge of the model's strengths and limitations. Like any good tool.

Re:Llama.cpp 30B runs with only 6GB of RAM now by mx20 on Saturday April 01, 2023 @05:19AM (#35398659)
Maybe a lot of people/companies also don't want to give their data and knowledge to OpenAI, so that they can sell it off to the competition.
Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Saturday April 01, 2023 @04:42PM (#35403868)
Yes, that's the "strategic" play I mentioned before.

This isn't really helpful for people who want open AI though, because if your strategy is to deny OpenAI data and knowledge then you aren't going to release any models either.

Re:Llama.cpp 30B runs with only 6GB of RAM now by chii on Friday March 31, 2023 @09:59PM (#35396273)
> Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication.

this is why i think the patent and copyright system is a failure. The idea that having laws protecting information like this would advance the progress of science.

It doesn't, because look how an illegally leaked model gets much more advances in shorter time. The laws protecting IP merely gives a moat to incumbents.

Re:Llama.cpp 30B runs with only 6GB of RAM now by hotpotamus on Friday March 31, 2023 @11:40PM (#35396877)
The idea of a patent was always a time-limited monopoly and in exchange you would reveal trade secrets that could presumably advance science. I think like many aspects of modernity, it's a bit outmoded these days, particularly in software, but that was the idea. Copyright was similar, but it did not last nearly as long as it does today in the original US incarnations.
Re:Llama.cpp 30B runs with only 6GB of RAM now by breck on Saturday April 01, 2023 @11:07AM (#35400870)
> The laws protecting IP merely gives a moat to incumbents.

Yes. These laws are bad. We could fix this with a 2 line change:

    Section 1. Article I, Section 8, Clause 8 of this Constitution is hereby repealed.
    Section 2. Congress shall make no law abridging the right of the people to publish information.
Re:Llama.cpp 30B runs with only 6GB of RAM now by kmeisthax on Saturday April 01, 2023 @12:35PM (#35401622)
Abolishing the copyright clause would not solve this problem because OpenAI is not leveraging copyright or patents. They're just not releasing anything.

To fix this, you'd need to ban trade secrecy entirely. As in, if you have some kind of invention or creative work you must publish sufficient information to replicate it "in a timely manner". This would be one of those absolutely insane schemes that only a villain in an Ayn Rand book would come up with.

Re:Llama.cpp 30B runs with only 6GB of RAM now by breck on Saturday April 01, 2023 @05:37PM (#35404345)
> Abolishing the copyright clause would not solve this problem because OpenAI is not leveraging copyright or patents. They're just not releasing anything.

The problem is how in the world is ChatGPT so good compared to the average human being? The answer is that human beings (except for the 1%), have their left hands tied behind their back because of copyright law.

Re:Llama.cpp 30B runs with only 6GB of RAM now by JBorrow on Saturday April 01, 2023 @09:36AM (#35400202)
$n00k of compute time is nothing, sorry. This is the kind of thing that academic institutions can give out for free…
Re:Llama.cpp 30B runs with only 6GB of RAM now by linesinthesand on Saturday April 01, 2023 @09:46AM (#35400271)
why don't you write the check then huh
Re:Llama.cpp 30B runs with only 6GB of RAM now by jart on Saturday April 01, 2023 @10:40PM (#35406593)
* * *
Re:Llama.cpp 30B runs with only 6GB of RAM now by barbariangrunge on Saturday April 01, 2023 @11:17AM (#35400942)
Can the foss community really find nobody with the motivation to use their Bitcoin rig as a research machine? Or do you need even more specialized hardware than that?
Re:Llama.cpp 30B runs with only 6GB of RAM now by stolsvik on Saturday April 01, 2023 @11:56AM (#35401274)
The bitcoin rig is specialised - it can only compute SHA hashes. You need more general compute.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Robotbeat on Friday April 14, 2023 @03:04PM (#35573426)
Training via CPU isn’t that bad if fully optimized with AVX512 extensions.
Re:Llama.cpp 30B runs with only 6GB of RAM now by datadeft on Friday March 31, 2023 @05:04PM (#35393627)
I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @05:06PM (#35393655)
You mean in contained app? It can already run on a phone. GPU acceleration would be nice at this point, though.
Re:Llama.cpp 30B runs with only 6GB of RAM now by datadeft on Sunday April 02, 2023 @02:52AM (#35407867)
I was thinking about the limitation by the mobile HW. Yeah GPU support would be nice.
Re:Llama.cpp 30B runs with only 6GB of RAM now by rickrollin on Friday March 31, 2023 @05:06PM (#35393662)
People have actually ran it on phones.
Re:Llama.cpp 30B runs with only 6GB of RAM now by datadeft on Sunday April 02, 2023 @02:52AM (#35407869)
This is what my comment is about. It happened much sooner than I thought.
Re:Llama.cpp 30B runs with only 6GB of RAM now by politician on Friday March 31, 2023 @05:25PM (#35393885)
Roughly: OpenAIs don’t employ enough jarts.

In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas.

When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.

Re:Llama.cpp 30B runs with only 6GB of RAM now by bee_rider on Friday March 31, 2023 @05:32PM (#35393957)
The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.
Re:Llama.cpp 30B runs with only 6GB of RAM now by fallous on Friday March 31, 2023 @07:47PM (#35395341)
The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.
Re:Llama.cpp 30B runs with only 6GB of RAM now by hedgehog on Friday March 31, 2023 @06:00PM (#35394249)
They have very good people but those people have other priorities.
Re:Llama.cpp 30B runs with only 6GB of RAM now by dsjoerg on Friday March 31, 2023 @07:04PM (#35394957)
Re:Llama.cpp 30B runs with only 6GB of RAM now by sp332 on Friday March 31, 2023 @07:18PM (#35395087)
In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America

https://en.m.wikipedia.org/wiki/Justine_Tunney

Re:Llama.cpp 30B runs with only 6GB of RAM now by hackernewds on Friday March 31, 2023 @09:44PM (#35396194)
what's your point? also interestingly JART is in the thread here, so they might have read your comment :)
Re:Llama.cpp 30B runs with only 6GB of RAM now by sp332 on Saturday April 01, 2023 @12:59PM (#35401864)
I'm pretty sure she knows already?
Re:Llama.cpp 30B runs with only 6GB of RAM now by dflock on Friday March 31, 2023 @10:39PM (#35396490)
The jart: the standard unit of developer.

"I've got 20 yrs experience, and I think I'm about 150 milli jarts, maybe 200 on a good day."

Re:Llama.cpp 30B runs with only 6GB of RAM now by fy20 on Saturday April 01, 2023 @03:13AM (#35397974)
I'd say it's probably not a priority for them right now.

Of course it would save them some money if they could run their models on cheaper hardware, but they've raised $11B so I don't think that's much of a concern right now. Better to spend the efforts on pushing the model forward, which some of these optimisations may make harder.

Re:Llama.cpp 30B runs with only 6GB of RAM now by VernorVintage on Saturday April 01, 2023 @03:55PM (#35403525)
It's a pretty big concern if you had to spend a billion on training, but 6 months later the open source community is able to replicate your training for <100K because you were too cheap to hire an extra 100 optimization experts

That'd be a 10,000 fold depreciation of an asset due to a preventable oversight. Ouchies.

Re:Llama.cpp 30B runs with only 6GB of RAM now by xienze on Friday March 31, 2023 @06:46PM (#35394786)
> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

It’s several things:

* Cutting-edge code, not overly concerned with optimization

* Code written by scientists, who aren’t known for being the world’s greatest programmers

* The obsession the research world has with using Python

Not surprising that there’s a lot of low-hanging fruit that can be optimized.

Re:Llama.cpp 30B runs with only 6GB of RAM now by Miraste on Friday March 31, 2023 @06:57PM (#35394897)
Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
Re:Llama.cpp 30B runs with only 6GB of RAM now by chatmasta on Friday March 31, 2023 @07:10PM (#35395016)
It might not be slow in general, but it's easy to write slow code in it.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Closi on Saturday April 01, 2023 @01:58AM (#35397547)
In ML python is effectively the interface rather than the bit that is doing the heavy lifting.

The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).

Re:Llama.cpp 30B runs with only 6GB of RAM now by chaorace on Friday March 31, 2023 @07:54PM (#35395401)
Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Max-Limelihood on Monday April 03, 2023 @06:57PM (#35432517)
In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.
Re:Llama.cpp 30B runs with only 6GB of RAM now by Max-Limelihood on Tuesday April 04, 2023 @12:42PM (#35442394)
It really is that slow.

You're completely correct that the speed-sensitive parts are written in lower-level libraries, but another way to phrase that is "Python can go really fast, as long as you don't use Python." But this also means ML is effectively hamstrung into only using methods that already exist and have been coded in C++, since anything in Python would be too slow to compete.

There's lots of languages that make good tradeoffs between performance and usability. Python is not one of those languages. It is, at best, only slightly harder to use than Julia, yet orders-of-magnitude slower.

Re:Llama.cpp 30B runs with only 6GB of RAM now by Closi on Saturday April 01, 2023 @01:56AM (#35397540)
I’m not sure this is fair - a lot of the performance optimisations have come from applied mathematicians rather than programmers, and python is not generally considered to be the bottleneck (it is the interface rather than what is running the computation - it calls a C API which then often uses CUDA and may also run on hardware specifically designed for ML).
Re:Llama.cpp 30B runs with only 6GB of RAM now by lannisterstark on Saturday April 01, 2023 @12:10AM (#35397029)
And THIS is why I always advocate for democratization of AI models, getting these into hands of open source community and letting people use it as they wish etc.

But a lot of people would rather only have govt or corp control of it...