News for hackers. Stuff that matters.
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by capableweb on Sunday April 30, 2023 @01:02PM (#35764468)
In my experience, heavy number crunching is more suitable to run on dedicated machines, rather than virtualized "cloud" cores. More consistent performance, no noisy neighbors and cheaper in the long term.
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by halflings on Sunday April 30, 2023 @01:19PM (#35764606)
> cheaper in the long term

Citation needed :)) There are economies of scale and various optimizations that are just not possible with dedicated machines.

Anecdotal evidence: a company I worked in had a dedicated DC with hundreds/thousands of machines that mostly ran SQL queries on petabytes of data (any query would take ~5-30 minutes). Eye-watering budget and whole teams to maintain the cluster... They switched to GCP/BigQuery, got queries that ran in seconds at a fraction of the budget.

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by slimsag on Sunday April 30, 2023 @03:13PM (#35765680)
The inverse is also true; I don't have to pay Google or AWS to deploy my software to dedicated consumer devices. The user already paid for it.
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by Negitivefrags on Sunday April 30, 2023 @03:36PM (#35765889)
This anecdote is the opposite of the ones I usually hear.

Economies of scale go in the other direction usually.

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by taneq on Sunday April 30, 2023 @08:12PM (#35767755)
Those economies are for the cloud provider, not you. The cost to you is set at just less than their best guess at what you think self-hosting will cost you.

Cloud hosting only makes sense in a case that your capacity needs are changing rapidly and unpredictably, or the case where you’re so big that the cloud hosting company is effectively a department. Any other time it will almost certainly be cheaper to self host.

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by rvnx on Sunday April 30, 2023 @07:33PM (#35767540)
Sounds like they could have switched to Clickhouse if they already had the hardware.
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by david-gpu on Sunday April 30, 2023 @02:12PM (#35765093)
"The cloud" simply means remote servers. Those servers may or may not contain application-specific hardware acceleration.

Nowadays many cell phone application processors also have dedicated hardware to accelerate neural nets, but they will always be limited by thermal constraints.

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by smoldesu on Sunday April 30, 2023 @02:13PM (#35765115)
If you can saturate it, sure. The average person is not going to be running 24/7 AI compute tasks though, so paying for a month of A100 dedi when you only need 10-20 minutes of compute/week is a complete waste.