News for hackers. Stuff that matters.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by kiratp on Friday June 14, 2024 @01:23PM (#40682755)
Nvidia EULA prevents you from using consumer gaming GPUs in datacenters so 4xxx cards are a non-starter for any service usecases

EDIT: TOS -> EULA per comments below

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by J_Shelby_J on Friday June 14, 2024 @01:29PM (#40682812)
What about on prem? Like, my small business needs an LLM. Can I put a 3090 in a box in a closet?

What if I’m a business and I’m selling LLMs in a box for you to put on a private network?

What constitutes a data center according to the ToS? Is it enforceable if you never agree to the ToS (buying through eBay?)

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by kiratp on Friday June 14, 2024 @01:36PM (#40682890)
By using the drivers you agree to their TOS. So yes, it applies even on your private network.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by swatcoder on Friday June 14, 2024 @01:52PM (#40683086)
The customer limitation described in the EULA is exactly this:

> No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.

- https://www.nvidia.com/content/DriverDownloads/licence.php?l...

There's no further elaboration on what "datacenter" means here, and it's a fair argument to say that a closet with one consumer-GPU-enriched PC is not a "datacenter deployment". The odds that Nvidia would pursue a claim against an individual or small business who used it that way is infinitesimal.

So both the ethical issue (it's a fair-if-debatable read of the clause) and the practical legal issue (Nvidia wouldn't bother to argue either way) seem to say one needn't worry about.

The clause is there to deter at-scale commercial service providers from buying up the consumer card market.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by light_hue_1 on Friday June 14, 2024 @01:43PM (#40682970)
Don't listen to this person. They have no idea what they're talking about.

No one cares about this TOS provision. I know both startups and large businesses that violate it as well as industry datacenters and academic clusters. There are companies that explicitly sell you hardware to violate it. Heck, Nvidia will even give you a discount when you buy the hardware to violate it in large enough volume!

You do you.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by wongarsu on Friday June 14, 2024 @02:13PM (#40683316)
In a previous AI wave hosters like OVH and Hetzner started offering servers with GTX 1080 at prices other hosters with datacenter-grade GPUs couldn't possibly compete with - and VRAM wasn't as big of a deal back then. That's who this clause targets.

If you don't rent our servers or VMs Nvidia doesn't care. They aren't Oracle.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by jtriangle on Friday June 14, 2024 @02:57PM (#40683809)
There are no nvidia police, they literally cannot stop you from doing this.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by nubinetwork on Friday June 14, 2024 @01:27PM (#40682793)
That never stopped the crypto farmers...
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by byteknight on Friday June 14, 2024 @01:28PM (#40682801)
They also weren't selling the usage of the cards.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by giancarlostoro on Friday June 14, 2024 @01:36PM (#40682902)
It's not in a data center, it's in his home.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by badgersnake on Friday June 14, 2024 @02:48PM (#40683707)
How would they even know?
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by oneshtein on Friday June 14, 2024 @01:28PM (#40682799)
Nvidia terms of what?
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by codetrotter on Friday June 14, 2024 @01:48PM (#40683030)
Parent commenter used the wrong word. It’s the EULA that prevents it.

Regardless, it is true that it is a problem.

https://www.reddit.com/r/MachineLearning/comments/ikrk4u/d_c...