News for hackers. Stuff that matters.
Re:Jamesob's guide to running SOTA LLMs locally by Der_Einzige on Friday July 03, 2026 @04:48PM (#48779785)
Use modern samplers and you don’t need to limit yourself to 8bit at half the context window. I could push it down to 1.58 bits and get decently good output easily by simply not using the garbage default top_p and top_k that vendors continue to wrongly recommend.
Re:Jamesob's guide to running SOTA LLMs locally by anon373839 on Friday July 03, 2026 @05:39PM (#48780321)
Where do you find optimal samplers and sampler settings for these models? Very interested in this as I, too, use Q8.