Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on ChatGPT loses users for first time, shaking faith in AI revolution. Parent comment
If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b.
If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU.
Ex - here's my nvidia-smi output while 65b is running
https://imgur.com/a/JnaieKg
There are no comments on this comment yet.
If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b.
If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU.
Ex - here's my nvidia-smi output while 65b is running
https://imgur.com/a/JnaieKg