News for hackers. Stuff that matters.
Re:Web LLM – WebGPU Powered Inference of Large Language Models by cahoot_bird on Saturday April 15, 2023 @04:06PM (#35584057)
Do LLMs have a way around the high end GPU requirements, or can CPU code potentially be much more optimized somehow?

This is the only thing I can think of, not everyone will have the latest high end GPUs to run such software..

Re:Web LLM – WebGPU Powered Inference of Large Language Models by m1el on Saturday April 15, 2023 @04:21PM (#35584178)
If you're doing inference on neural networks, each weight has to be read at least once per token. This means you're going to read at least the size of the entire model, per token, at least once during inference. If your model is 60GB, and you're reading it from the hard drive, then your bare minimum time of inference per token will be limited by your hard drive read throughput. Macbooks have ~4GB/s sequential read speed. Which means your inference time per token will be strictly more than 15 seconds. If your model is in RAM, then (according to Apple's advertising) your memory speed is 400GB/s, which is 100x your hard drive speed, and just the memory throughput will not be as much of a bottleneck here.
Re:Web LLM – WebGPU Powered Inference of Large Language Models by rahimnathwani on Saturday April 15, 2023 @04:54PM (#35584428)
Your answer applies equally to GPU and CPU, no?

The comment to which you replied was asking about the need for a GPU, not the need for a lot of RAM.

Re:Web LLM – WebGPU Powered Inference of Large Language Models by brucethemoose2 on Saturday April 15, 2023 @04:29PM (#35584241)
The next generation of Intel/AMD IGPs operating out of RAM should be quite usable.
Re:Web LLM – WebGPU Powered Inference of Large Language Models by verdverm on Saturday April 15, 2023 @04:09PM (#35584091)
Check out LLAMA-CPP
Re:Web LLM – WebGPU Powered Inference of Large Language Models by cahoot_bird on Saturday April 15, 2023 @06:01PM (#35585024)
Looks like this was hacked together pretty quickly. This in CPU is exactly what needs optimized to run on more devices, if that's even possible..

I guess it will take hardware and software a while to catch up to compete with ChatGPT..

Re:Web LLM – WebGPU Powered Inference of Large Language Models by verdverm on Saturday April 15, 2023 @06:59PM (#35585507)
If you look at the news, yes it came together quickly, but it has also gotten a lot of performance upgrades which have made significant improvements.
Re:Web LLM – WebGPU Powered Inference of Large Language Models by ratg13 on Saturday April 15, 2023 @06:23PM (#35585207)
There will be LLM specific chips coming to market soon which will be specialized to the task.

Tesla already has already been creating AI chips for their FSD features in their vehicles. Over the next years, everyone will be racing to be the first to put out LLM specific chips, with AI specific hardware devices following.

Re:Web LLM – WebGPU Powered Inference of Large Language Models by dan_mctree on Sunday April 16, 2023 @01:19AM (#35587473)
What exactly is the ideal sort of hardware to be able to run and train large models? Do you basically just need a high end version of basically everything?