Re:Ask HN: Affordable hardware for running local large langu...
by s1gsegv on Sunday May 05, 2024 @08:25PM
(#40269915)
Yes, Metal seems to allow a maximum of 1/2 of the RAM for one process, and 3/4 of the RAM allocated to the GPU overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182
[ Parent
| on HN ]
There are no comments on this comment yet.