News for hackers. Stuff that matters.
Re:Localllm lets you develop gen AI apps on local CPUs by api on Wednesday February 07, 2024 @06:15PM (#39295622)
You could buy 4-6 machines like that per year at that price. Gives you a taste of cloud mark-up. Bandwidth markup is far more insane.
Re:Localllm lets you develop gen AI apps on local CPUs by eyegor on Wednesday February 07, 2024 @08:38PM (#39296789)
For $1000 you could basically build one today. 5950x (16c cpu) is $400, 128gb kit of ddr4 is $250-300, maybe enough budget left for mobo, ssd, psu, case, cooler if you bought used bits (may also need cheapest gpu on Craigslist depending on your needs). All new ~$1200. Aws/gcp prices are nuts compared to on prem, although hetzner and co would be much cheaper.
Re:Localllm lets you develop gen AI apps on local CPUs by omgwtfbyobbq on Thursday February 08, 2024 @12:23AM (#39298556)
Yeah, Cloud is definitely a money maker.

If someone has a microcenter nearby, they could build an 8 core ddr4 system with 128gb of ram for maybe $800.

$3200+ could build a 64 core/128gb ddr5 rig.

https://www.newegg.com/tyan-s8050gm2ne-9534-amd-epyc-9534-2-...

Re:Localllm lets you develop gen AI apps on local CPUs by RecycledEle on Thursday February 08, 2024 @03:57PM (#39307587)
Price a HP z440 with those specs on eBay or Amazon.
Re:Localllm lets you develop gen AI apps on local CPUs by mkatx on Wednesday February 07, 2024 @08:44PM (#39296821)
Yep, I built gaming PCs for years, and I have a few laying around.. Add some ram and they a great servers (without memory safety though).
Re:Localllm lets you develop gen AI apps on local CPUs by api on Thursday February 08, 2024 @07:20AM (#39301084)
Without memory safety? Do you mean ECC?

Non-ECC issues are statistically rare. You might never see them if you aren’t running a lot of systems at heavy load at scale.

Re:Localllm lets you develop gen AI apps on local CPUs by mkatx on Thursday February 08, 2024 @06:01PM (#39309000)
Yeah, ecc. I do have a lot of data intensive services running. But my systems are mostly amd, and I'm pretty sure my amd CPUs can actually use ecc memory, so I'll probably upgrade some day.
Re:Localllm lets you develop gen AI apps on local CPUs by api on Thursday February 08, 2024 @06:27PM (#39309246)
I've run non-ECC systems at heavy load for uptimes in excess of a year and not had issues. It really is statistically rare... but of course common enough that if you run whole data centers you will see it.