News for hackers. Stuff that matters.
Re:Show HN: Finetune LLaMA-7B on commodity GPUs using your o... by albertzeyer on Wednesday March 22, 2023 @05:02AM (#35258531)
The attention implementation is in a way that the memory scales quadratically with sequence lengths. Overall, this is still a small factor compared to just the model weights, but at some seq lengths, this would dominate.

By using flash attention, you can get memory requirement down to scale linearly with sequence lengths.

There are no comments on this comment yet.