Re:Llama.cpp: Full CUDA GPU Acceleration
by bartwr on Monday June 12, 2023 @11:01PM
(#36304627)
Even "compiled" JAX or PyTorch can leave some performance even if you hit the common path (It's also "praying" that the compiler actually works if you do anything non-standard).
But memory wise, there is almost no optimization or reuse (and it's sacrificed for performance), which leads to insane memory usage.
And it's not that they are bad - but optimal compilation of a graph is combinatorially explosive problem, impossible without heuristics and guesswork (what to reuse and waste memory vs recompute). A good programmer can do a significantly better job.
But memory wise, there is almost no optimization or reuse (and it's sacrificed for performance), which leads to insane memory usage.
And it's not that they are bad - but optimal compilation of a graph is combinatorially explosive problem, impossible without heuristics and guesswork (what to reuse and waste memory vs recompute). A good programmer can do a significantly better job.