How to run Llama 13B with a 6GB graphics card

Re:How to run Llama 13B with a 6GB graphics card by instance on Sunday May 14, 2023 @10:16AM (#35938173)

I tested on a serious use case and quality was subpar. For real use cases I had to either host the most powerful model you can get (e.g. LLaMA-65B or so) on a cloud machine, which again costs too much (you'll be paying like 500-1000 USD per month), or just go straight for GPT-3.5 on OpenAI. The latter economically makes most sense.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by inferense on Sunday May 14, 2023 @12:33PM (#35939173)

what real use case did you use it for?

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by throwaway1777 on Sunday May 14, 2023 @12:37PM (#35939207)

Making demos to raise investment probably

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by instance on Sunday May 14, 2023 @01:35PM (#35939678)

For instance used it in conjunction with llama-index for knowledge management. Created an index for a whole confluence/jira of a mid-sized company, got good results with GPT, but for LLaMA of this size that use case was too much.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by sroussey on Sunday May 14, 2023 @05:24PM (#35941454)

Did you try instructor-xl? It ranks highest on huggingface.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by dzhiurgis on Sunday May 14, 2023 @05:26PM (#35941468)

I'd argue 1k per month for mid-sized company is nothing, but I can understand where you are coming from.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by raffraffraff on Sunday May 14, 2023 @01:44PM (#35939767)

What about turning the cloud vm off except when you're actually using it?

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by unglaublich on Sunday May 14, 2023 @03:06PM (#35940468)

A "serious use case" means it needs to be available around the clock.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by quickthrower2 on Sunday May 14, 2023 @07:31PM (#35942314)

So modal.com is "turning-the-vm-off-when-unused-as-a-service" :-)

I ran research/open_llama_7b_preview_200bt on there, using they python example, with A10G gpu.

Cost 2-3c per run, taking ~20 seconds each time, on fairly small prompts. So about the same as GPT-4?

Now this is a non expert just playing, it probably can be optimized by trying different GPUs and optimizing the code somehow.

I don't think you are using these models to save money, but you might be using them for tunability, privacy, mobility [1], secrecy or fun/research.

[1] in other words you want to build a robot that can work disconnected from the internet.

[ Parent | on HN ]