News for hackers. Stuff that matters.

A comment on How to run Llama 13B with a 6GB graphics card. Parent comment

Re:How to run Llama 13B with a 6GB graphics card by MPSimmons on Sunday May 14, 2023 @11:54PM (#35943724)
How do you deal with the 'too many tokens' problem? Like, For 13b, max tokens I've gotten it to is 508 - is there a way to increase the number it will accept?
[ Parent | on HN ]
Re:How to run Llama 13B with a 6GB graphics card by MPSimmons on Monday May 15, 2023 @12:42AM (#35943953)
Oh, I figured this out. It runs with 512 as the default context, but you can increase it with -c
[ Parent | on HN ]