News for hackers. Stuff that matters.

A comment on 70B Llama 2 at 35tokens/second on 4090. Parent comment

Re:70B Llama 2 at 35tokens/second on 4090 by redox99 on Wednesday September 13, 2023 @05:37AM (#37494428)
That's absolutely false, unless you mean running llamacpp with half the layers on the CPU and half the layers on the GPU. And that was like 2t/s.
[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by yumraj on Wednesday September 13, 2023 @11:40AM (#37498234)
> running llamacpp with half the layers on the CPU and half the layers on the GPU

How do you do that? Could you please provide a link if you have it handy.

[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by ynniv on Wednesday September 13, 2023 @02:31PM (#37499947)
When running you use "-ngl" to specify how many layers are uploaded to the GPU
[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by gsuuon on Wednesday September 13, 2023 @02:33PM (#37499978)
Just set the --gpu-layers|-ngl flag. Run main with --help to see all the options.
[ Parent | on HN ]