News for hackers. Stuff that matters.
Re:The Coming of Local LLMs by binkHN on Tuesday April 11, 2023 @11:55AM (#35526961)
This is wonderful. As hardware and software continues to improve, everything seems to find a way to run on ever smaller devices. Guess your own pocket-AGI is not too far away after all.
Re:The Coming of Local LLMs by 1827162 on Tuesday April 11, 2023 @12:54PM (#35527824)
By the way, I was thinking of something along the lines of a powerful FPGA with direct access to large quantities of very fast NAND flash, likely many chips in parallel, which will save having to load the model into RAM..... So it will be able to directly run from NAND flash, which opens up the possibility of using very large models???

Power consumption would not be an issue if it's used sporadically throughout the day, it's not like it needs to run continuously?

There is still the issue of NAND flash read disturb, which I haven't fully looked into yet.