News for hackers. Stuff that matters.
Re:I Self-Hosted Llama 3.2 with Coolify on My Home Server by khafra on Wednesday October 16, 2024 @03:49AM (#41856609)
If you want to set up an AI server for your own use, it's exceedingly easy to install LM Studio and hit the "serve an API" button.

Testing performance this way, I got about 0.5-1.5 tokens per second with an 8GB 4bit quantized model on an old DL360 rack-mount server with 192GB RAM and 2 E5-2670 CPUs. I got about 20-50 tokens per second on my laptop with a mobile RTX 4080.

Re:I Self-Hosted Llama 3.2 with Coolify on My Home Server by taosx on Wednesday October 16, 2024 @04:02AM (#41856694)
LM studio is so nice, I'm up and running in 5 minutes. ty