News for hackers. Stuff that matters.

A comment on Ask HN: How do you scale transformer context lengths over multiple machines?.

Re:Ask HN: How do you scale transformer context lengths over... by brucethemoose2 on Friday November 24, 2023 @02:34AM (#38401448)
Quantizing it down to 8 bits seems to be one solution. TensorRT-LLM does this (and I think requires an H100)? exLlama also does this on much lesser hardware.
[ Parent | on HN ]
Re:Ask HN: How do you scale transformer context lengths over... by mlstudies on Friday November 24, 2023 @02:51AM (#38401548)
wouldn't that mean trying to fit it on one machine?
[ Parent | on HN ]
Re:Ask HN: How do you scale transformer context lengths over... by brucethemoose2 on Friday November 24, 2023 @03:04AM (#38401609)
Indeed :P

Honestly I'm not sure how context "sharding" works on multiple GPUs atm. Decent, really long context OSS models like Yi 200K and YARN finetunes are very new.

[ Parent | on HN ]