News for hackers. Stuff that matters.

A comment on Serving AI from the Basement – 192GB of VRAM Setup. Parent comment

Re:Serving AI from the Basement – 192GB of VRAM Setup by nathanasmith on Thursday September 12, 2024 @10:17AM (#41521292)
The system has 512 GB of RAM so while it'll be slower at inference, he really has about 704 GB at his disposal to run the model assuming he distributes the weights across the VRAM and system RAM.
[ Parent | on HN ]

There are no comments on this comment yet.