News for hackers. Stuff that matters.
Re:Run llama3 locally with 1M token context by ukuina on Tuesday April 30, 2024 @07:20PM (#40217669)
Tested this on Apple Silicon at 128k context, very poor results with random character hallucinations.
Re:Run llama3 locally with 1M token context by growfeather on Wednesday May 01, 2024 @12:47AM (#40219602)
Thanks for testing it. I wonder if there’s a reasonable line? 64k? How much RAM did the 128k context require? (Maybe future models might be better at that level)
Re:Run llama3 locally with 1M token context by nicognaw on Wednesday May 01, 2024 @01:07AM (#40219708)
If you got random tokens on Apple Silicon Macs, you should check your stderr for Metal’s error code, like 5 for out of metal memory. Metal lets you use half of your unified memory as metal memory.