Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by Taek on Wednesday March 22, 2023 @01:29AM
(#35257190)
Looks like you need about 120 GB to fine-tune the 65B model with this code at a sequence length of 512. How does the memory usage scale as the sequence length grows?
[ Parent
| on HN ]
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by lxe on Wednesday March 22, 2023 @01:38AM
(#35257230)
> How does the memory usage scale as the sequence length grows?
That's a good question. I was under the assumption it's linearly proportional, but I can test it out I guess.
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by Taek on Wednesday March 22, 2023 @01:53AM
(#35257320)
I suspect its linear with a small constant factor.
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by lxe on Wednesday March 22, 2023 @01:32AM
(#35257203)
Lots of VRAM, but the method is the same. Here's someone who finetuned llama-30b on alpaca dataset for example: https://github.com/deep-diver/Alpaca-LoRA-Serve
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by arpowers on Wednesday March 22, 2023 @02:18AM
(#35257471)
Really takes that much vram??
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by ioedward on Wednesday March 22, 2023 @04:21AM
(#35258242)
Normally people split up the model across multiple GPUs, i.e. model/tensor parallelism.
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o...
by albertzeyer on Wednesday March 22, 2023 @05:02AM
(#35258531)
The attention implementation is in a way that the memory scales quadratically with sequence lengths. Overall, this is still a small factor compared to just the model weights, but at some seq lengths, this would dominate.
By using flash attention, you can get memory requirement down to scale linearly with sequence lengths.