How to run Llama 13B with a 6GB graphics card

Re:How to run Llama 13B with a 6GB graphics card by elorant on Sunday May 14, 2023 @12:24PM (#35939102)

There's an online demo of Vicuna-13b where you can test its efficiency:

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by csdvrx on Sunday May 14, 2023 @12:50PM (#35939303)

Yes, but can you replicate that functionality using llama.cpp?

If so, what did you run with main?

I haven't been able to get an answer, while for the question above, I can get 'I chose the word "laptop"' with mlc-llm

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by elorant on Sunday May 14, 2023 @04:29PM (#35941117)

For the tasks I need it the efficiency is similar to the online model. Only slower. I don't care for conversational functionality.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by techload on Sunday May 14, 2023 @06:04PM (#35941740)

After two prompts I was astounded by the innacuracies present in the answers. An they were pretty easy questions.

[ Parent | on HN ]