News for hackers. Stuff that matters.

federicoTXTS

Show HN: I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp

https://github.com/FedericoTs/quantprobe

Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen...

( Read More... | no comments | on HN )