News for hackers. Stuff that matters.
Re:Ask HN: Affordable hardware for running local large langu... by s1gsegv on Sunday May 05, 2024 @08:25PM (#40269915)
Yes, Metal seems to allow a maximum of 1/2 of the RAM for one process, and 3/4 of the RAM allocated to the GPU overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182

There are no comments on this comment yet.