You can also get exactly that same VM as a spot instance with a 10GB disk and external ip address in the us-west4 region for $142.04/month. (Spot instance prices vary dramatically from one region to another, but that one is very discounted in us-west4.)

Also, is this localllm kind of like a Google competitor to some aspects of ollama?

You could buy 4-6 machines like that per year at that price. Gives you a taste of cloud mark-up. Bandwidth markup is far more insane.

For $1000 you could basically build one today. 5950x (16c cpu) is $400, 128gb kit of ddr4 is $250-300, maybe enough budget left for mobo, ssd, psu, case, cooler if you bought used bits (may also need cheapest gpu on Craigslist depending on your needs). All new ~$1200. Aws/gcp prices are nuts compared to on prem, although hetzner and co would be much cheaper.

Yeah, Cloud is definitely a money maker.

If someone has a microcenter nearby, they could build an 8 core ddr4 system with 128gb of ram for maybe $800.

$3200+ could build a 64 core/128gb ddr5 rig.

https://www.newegg.com/tyan-s8050gm2ne-9534-amd-epyc-9534-2-...

Price a HP z440 with those specs on eBay or Amazon.

Yep, I built gaming PCs for years, and I have a few laying around.. Add some ram and they a great servers (without memory safety though).

Without memory safety? Do you mean ECC?

Non-ECC issues are statistically rare. You might never see them if you aren’t running a lot of systems at heavy load at scale.

Yeah, ecc. I do have a lot of data intensive services running. But my systems are mostly amd, and I'm pretty sure my amd CPUs can actually use ecc memory, so I'll probably upgrade some day.

I've run non-ECC systems at heavy load for uptimes in excess of a year and not had issues. It really is statistically rare... but of course common enough that if you run whole data centers you will see it.