News for hackers. Stuff that matters.
Re:Using LLaMA with M1 Mac and Python 3.11 by flangola7 on Sunday March 12, 2023 @03:58PM (#35125412)
Each token is inferenced against the entire model. For the largest model that means 60GB of data or at least 10 seconds per token on the fastest SSDs. Very heavy SSD wear from that many read operations would quickly burn out even enterprise drives too.
Re:Using LLaMA with M1 Mac and Python 3.11 by orf on Sunday March 12, 2023 @04:03PM (#35125481)
SSDs don’t wear from reading, only from writing.

Assuming a sensible, somewhat linear layout using mmap to map the weights would give you the ability to load a lot in memory, with potentially a fairly minimal page-in overhead