GPU is massively faster than CPUs for this sort of work load.

Even if you could use all 128 cores I would still get the 2080 would be faster

ah ok, that's fair.

Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot:

https://www.pcworld.com/article/402592/nvidia-turing-gpu-gef...

IMHO these chips designed for specialized workloads are looking more and more ridiculous. I expect GPU manufacturers to keep dragging them out for another decade or so, as well as Apple as it explores offshoots of M1. It all makes me feel very tired (the ultimate code smell).

A better design would be something like a 256+ core RISC-V with local memories in/by each core for data-locality and a content-addressable caching scheme for deduplication. Copy-on-write languages like Clojure and orchestrating processes under something like Docker would make it a breeze to program, although it would still support manually managed mutability like with Rust for innermost loops in games or whatever. It's fairly obvious how it would all work, but IP law and gatekeeping ensure that it will not happen anytime soon.

Then stuff like CUDA would be just another framework run on a symmetric multiprocessor and we could get back to exploring alternatives like genetic algorithms like we did in the 1990s. Thankfully nobody cares what I think, heck even I'm sick of reading my own complaints, so it's easy enough to just unsee this.

> Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot

I mean, “many” is not usually the case; an RTX 3090 has 10,496 CUDA cores, 328 Tensor cores, 82 RT (raytracing) cores, and 96 render output pipelines. ML apps will uses the first and, depending on software, second set. The vast majority of the cores being CUDA cores is the norm.

This is not strictly true. GPUs fare better at these tasks for a few reasons:

* The largest contributor is the sheer number of cores.

* The latency between cores and memory.

* FP16 instructions.

128 cores isn't an insignificant fraction of the number of cores on a 1050 (about 600), but CPU cores are individually more powerful. That advantage is potentially difficult to call. The top-end of Genoa has 96/192 cores, and you can slot many of them onto a single board.

AMD is throwing more and more memory into the CPU cache. That's very different to a direct path to GBs of HBM, but at some point the difference in performance might not matter to a novice/dabbler.

I would say give it a go, see what happens.

Look into sparsification of ML models: https://neuralmagic.com/ If your cpu has avx512 it can make it even faster.

I'm using a Ryzen 9 5950X to run some tests with Whisper (ASR), and since I have no GPU with more than 4 GB VRAM, I'm running it on the CPU. It is slow. It takes between 20 seconds and 2 minutes to transcribe 1 minute of audio, I'm using 8 cores. Adding more cores doesn't seem to improve the inferencing time.

ML is really something which should be left to a GPU.

There is an example to use multi GPU in the link, Outside of this I have read about offloading to cpu/nvme in the case of 100GB+ models that don't fit in VRAM tho this will be at the expense of performance.

https://pypi.org/project/rwkvstic/