News for hackers. Stuff that matters.
Re:How to run Llama 13B with a 6GB graphics card by sp332 on Sunday May 14, 2023 @05:17PM (#35941409)
Most people are running these at 4 bits per parameter for speed and RAM reasons. That means the model would take just about all of the RAM. But instead of swap (writing data to disk and then reading it again later), I would expect a good implementation to only run into cache eviction (deleting data from RAM and then reading it back from disk later), which should be a lot faster and cause less wear and tear on SSDs.

There are no comments on this comment yet.