How to run Llama 13B with a 6GB graphics card

Re:How to run Llama 13B with a 6GB graphics card by csdvrx on Sunday May 14, 2023 @12:07PM (#35938984)

I think the previous comment is more interested in your experience with your large data: what are you doing with it?

I have similar hardware at home, so I wonder how reliably you can process simple queries using domain knowledge + logic which work on on mlc-llm, something like "if you can chose the word food, or the word laptop, or the word deodorant, which one do you chose for describing "macbook air"? answer precisely with just the word you chose"

If it works, can you upload the weights somewhere? IIRC, vicuna is open source.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by elorant on Sunday May 14, 2023 @12:24PM (#35939102)

There's an online demo of Vicuna-13b where you can test its efficiency:

https://chat.lmsys.org/

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by csdvrx on Sunday May 14, 2023 @12:50PM (#35939303)

Yes, but can you replicate that functionality using llama.cpp?

If so, what did you run with main?

I haven't been able to get an answer, while for the question above, I can get 'I chose the word "laptop"' with mlc-llm

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by elorant on Sunday May 14, 2023 @04:29PM (#35941117)

For the tasks I need it the efficiency is similar to the online model. Only slower. I don't care for conversational functionality.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by techload on Sunday May 14, 2023 @06:04PM (#35941740)

After two prompts I was astounded by the innacuracies present in the answers. An they were pretty easy questions.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by chaxor on Sunday May 14, 2023 @05:28PM (#35941487)

If these problems are all very similar in structure, then you may not need an LLM. Simple GloVe or W2V may suffice with a dot product. The. You can plow through a few terabytes by the time the LLM goes through a fraction of that.

[ Parent | on HN ]