Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by gpm on Monday March 13, 2023 @03:45PM (#35141376)

> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16

LLaMa weights are f16s to start out with, no lowering necessary to get to there.

You can stream weights from RAM to the GPU pretty efficiently. If you have >= 32GB ram and >=2GB vram my code here should work for you: https://github.com/gmorenz/llama/tree/gpu_offload

There's probably a cleaner version of it somewhere else. Really you should only need >= 16 GB ram, but the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously. You could also lower vram requirements a bit more with a bit more work (I just made the smallest change possible to make it work)

[ Parent | on HN ]

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by nextaccountic on Monday March 13, 2023 @09:40PM (#35146181)

> the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously

This is the kind of thing that the stable diffusion community optimized the shit out

[ Parent | on HN ]