How to run Llama 13B with a 6GB graphics card

Re:How to run Llama 13B with a 6GB graphics card by mycall on Sunday May 14, 2023 @11:30PM (#35943601)

I wonder if specialization of the LLM is another way to reduce the RAM requirements. For example, if you can tell which nodes are touched through billions of web searches on a topic, then you can delete the ones that never are touched.

[ Parent | on HN ]

Re:How to run Llama 13B with a 6GB graphics card by opyate on Monday May 15, 2023 @05:09AM (#35945479)

Kind of like "tree shaking" for weights? Like dead code elimination.

[ Parent | on HN ]