Have a 13B running on a 3070 with 16 gpu layers and the rest running off CPU.
Performs okay, but way cheaper than renting a GPU on the cloud.
brew install ollama; ollama serve; ollama pull llama3: 8b-v2.9-q5_K_M; ollama run llama3: 8b-v2.9-q5_K_M
https://ollama.com/library/dolphin-llama3:8b-v2.9-q5_K_M
(It may need to be Q4 or Q3 instead of Q5 depending on how the RAM shakes out. But the Q5_K_M quantization (k-quantization is the term) is generally the best balance of size vs performance vs intelligence if you can run it, followed by Q4_K_M. Running Q6, Q8, or fp16 is of course even better but you’re nowhere near fitting that on 8gb.)
https://old.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie...
Dolphin-llama3 is generally more compliant and I’d recommend that over just the base model. It's been fine-tuned to filter out the dumb "sorry I can't do that" battle, and it turns out this also increases the quality of the results (by limiting the space you're generating, you also limit the quality of the results).
https://erichartford.com/uncensored-models
https://arxiv.org/abs/2308.13449
Most of the time you will want to look for an "instruct" model, if it doesn't have the instruct suffix it'll normally be a "fill in the blank" model that finishes what it thinks is the pattern in the input, rather than generate a textual answer to a question. But ollama typically pulls the instruct models into their repos.
(sometimes you will see this even with instruct models, especially if they're misconfigured. When llama3 non-dolphin first came out I played with it and I'd get answers that looked like stackoverflow format or quora format responses with ""scores"" etc, either as the full output or mixed in. Presumably a misconfigured model, or they pulled in a non-instruct model, or something.)
Dolphin-mixtral:8x7b-v2.7 is where things get really interesting imo. I have 64gb and 32gb machines and so far the Q6 and q4-k_m are the best options for those machines. dolphin-llama3 is reasonable but dolphin-mixtral is a richer better response.
I’m told there’s better stuff available now, but not sure what a good choice would be for for 64gb and 32gb if not mixtral.
Also, just keep an eye on r/LocalLLaMA in general, that's where all the enthusiasts hang out.
Just download a single file and run it.
It’s been an absolute gamechanger.
the docs are good. when creating the initial CA make absolutely sure you set the CA expiration to 10-30 years, the default is 1 which means your whole setup explodes in a year without warning.
It’s end to end encrypted, and with tail lock enabled, nodes can not be added without user’s permission.
I didn’t start with tailscale because the only way you could log into it was with Google or GitHub or something. I don’t trust Microsoft or Google with auth for my internal network. I thought about running Headscale but Nebula was faster/easier for me.
There's also potential for malicious updates to compromise a network (as there is with most software unless you're auditing the source for each update).
E2EE is only as meaningful as where the keys reside, and how easily those keys are abused.
The metadata is generally public information, I don’t care about that.
The malicious updates and key abuse are more concerning. It’s true for all software, and probably better done with OS, like on iOS.
The VPN could steal the keys, but that’s a lawsuit!
But with a malicious update, they could ship them to their infra, targeting some users. The product then becomes malware!
totally different use case.
I think port forwarding configuration is a pain that does not offer value over just poking a hole in your firewall to do an authenticated connection over ssh.
Ollama makes it dead simple just to try it out. I was pleasantly surprised by the tokens/sec I could get with Llama 3 8B on a 2021 M1 MBP. Now need to try on my gaming PC I never use. Would be super cool to just have a LLM server on my local network for me and the fam. Exciting times.
I'm guessing actual break-even time is less than half that, so maybe 2 years.
This whole post makes OpenAI look like a better deal than it actually is.
I’ve gone through hundreds of millions, maybe billions, of tokens in the past year.
This article is just “cloud is expensive” 101. Nothing new.
I don't do any work with the output, I'm just the MLOps guy (ahem, DevOps).
You mention expense but on a purely financial basis I find any of these hosted solutions really hard to justify against GPT 3.5 turbo prices, including building your own rig. $5k + electricity is loads of 3.5 Turbo tokens.
Of course none of the data scientists or researchers I work with want to use that though - it's not their job to host these things or worry about the costs.
Knowing up front this is my fixed ML budget gives me peace of mind and gives me room to try stupid ideas without worrying about it.
Whereas doing it in the cloud you can a) get slammed with some crazy bill by accident, b) have to think more about what resources testing an idea will take, or conversely c) getting GPU FOMO and thinking “if just upgrade a level all my problems will be solved”.
It works for me, everybody mileage varies but personally I like to budget; spend; and then totally focus on my goals and not my cloud spend.
I’m also from the pre-cloud era, so doing stuff on my own bare metal is second nature.
I need to tell if I change I made was impactful, but if the model just magically gets smarter or dumber at my tasks with no warning then I can’t tell if I made an improvement or a regression.
Whereas the model on my GPU doesn’t change unless I change it. So it’s one less variable and LLM are black box to start with.
I may be wrong for Gemini, but my impression is all the companies are constantly tweaking the big models. I know GPT on Monday is not always the same GPT on Thursday for example.
I would not bank on anything in this article. It might as well have been written by a tiny Llama model.
EDIT: TOS -> EULA per comments below
What if I’m a business and I’m selling LLMs in a box for you to put on a private network?
What constitutes a data center according to the ToS? Is it enforceable if you never agree to the ToS (buying through eBay?)
> No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.
- https://www.nvidia.com/content/DriverDownloads/licence.php?l...
There's no further elaboration on what "datacenter" means here, and it's a fair argument to say that a closet with one consumer-GPU-enriched PC is not a "datacenter deployment". The odds that Nvidia would pursue a claim against an individual or small business who used it that way is infinitesimal.
So both the ethical issue (it's a fair-if-debatable read of the clause) and the practical legal issue (Nvidia wouldn't bother to argue either way) seem to say one needn't worry about.
The clause is there to deter at-scale commercial service providers from buying up the consumer card market.
No one cares about this TOS provision. I know both startups and large businesses that violate it as well as industry datacenters and academic clusters. There are companies that explicitly sell you hardware to violate it. Heck, Nvidia will even give you a discount when you buy the hardware to violate it in large enough volume!
You do you.
If you don't rent our servers or VMs Nvidia doesn't care. They aren't Oracle.
Regardless, it is true that it is a problem.
https://www.reddit.com/r/MachineLearning/comments/ikrk4u/d_c...
Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.
Add in $1,000 to setup the rest of the rig and you have a final price of around:
$2,800 + $1,000 = $3,800
This whole exercise assumes that you're using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you're willing to buy GPU hardware from eBay, that will cost significantly less.
I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.