News for hackers. Stuff that matters.

sssummer

Show HN: KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill

https://github.com/kvcache-ai/ktransformers

Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising...

( Read More... | no comments | 8 points | on HN )

Show HN: KTransformers–236B Model and 1M Context LLM Inference on Local Machines

https://github.com/kvcache-ai/ktransformers

Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with...

( Read More... | 3 comments | 6 points | on HN )

Older »