Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on 70B Llama 2 at 35tokens/second on 4090. Parent comment
Llama.cpp allows you to specify that n layers load into the GPU's VRAM, and the rest load into main memory.
There are no comments on this comment yet.
Llama.cpp allows you to specify that n layers load into the GPU's VRAM, and the rest load into main memory.