News for hackers. Stuff that matters.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by J_Shelby_J on Friday June 14, 2024 @01:29PM (#40682812)
What about on prem? Like, my small business needs an LLM. Can I put a 3090 in a box in a closet?

What if I’m a business and I’m selling LLMs in a box for you to put on a private network?

What constitutes a data center according to the ToS? Is it enforceable if you never agree to the ToS (buying through eBay?)

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by kiratp on Friday June 14, 2024 @01:36PM (#40682890)
By using the drivers you agree to their TOS. So yes, it applies even on your private network.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by swatcoder on Friday June 14, 2024 @01:52PM (#40683086)
The customer limitation described in the EULA is exactly this:

> No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.

- https://www.nvidia.com/content/DriverDownloads/licence.php?l...

There's no further elaboration on what "datacenter" means here, and it's a fair argument to say that a closet with one consumer-GPU-enriched PC is not a "datacenter deployment". The odds that Nvidia would pursue a claim against an individual or small business who used it that way is infinitesimal.

So both the ethical issue (it's a fair-if-debatable read of the clause) and the practical legal issue (Nvidia wouldn't bother to argue either way) seem to say one needn't worry about.

The clause is there to deter at-scale commercial service providers from buying up the consumer card market.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by light_hue_1 on Friday June 14, 2024 @01:43PM (#40682970)
Don't listen to this person. They have no idea what they're talking about.

No one cares about this TOS provision. I know both startups and large businesses that violate it as well as industry datacenters and academic clusters. There are companies that explicitly sell you hardware to violate it. Heck, Nvidia will even give you a discount when you buy the hardware to violate it in large enough volume!

You do you.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by wongarsu on Friday June 14, 2024 @02:13PM (#40683316)
In a previous AI wave hosters like OVH and Hetzner started offering servers with GTX 1080 at prices other hosters with datacenter-grade GPUs couldn't possibly compete with - and VRAM wasn't as big of a deal back then. That's who this clause targets.

If you don't rent our servers or VMs Nvidia doesn't care. They aren't Oracle.