om8
om8 on Hacker News
Existential risk can be another big reason
No, thanks, I have my own
Could've been better if GGUF implemented QTIP format. GGUF representation is a major limitation for llama.cpp quantization performance
> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
Why int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes.
> So how can I regain my ability to focus?
Getting diagnosed with ADHD and taking meds did the trick for me.
Not sure how relevant this is, since inability to focus was always an issue for me.
This project needs webgpu -- I did it on cpu about a year ago.
My demo uses 2 bit quantization to run llama3 models on any device with enough ram.
https://galqiwi.github.io/aqlm-rs/
> manufacturing companies making the flimsiest, cheapest, plastic crap to save 1/3 of a cent on every mop they produce. Designed to work for the least amount of time before needing replaced
We live in a world with such companies, and we can still buy quality things. If there is a demand for the purely-human generated texts, they will be around. Perhaps a lot of people around you will read ai text instead, and you'll get upset because of it, but it's their choice. You'll still have your thing
Oh, yes, you are right about watts. My bad
> volt-amperes are technically the same unit as watts
volt-amperes are joules
Cool way to think about GWh/year:
1 GWh/year = (10 ** 9) / 24 / 365.25 / (10 ** 6) MW = 0.11 MW
70 GWh/year = 8 MW
1755 GWh/year = 200 MW
252 GWh/year = 29 MW
cargo/uv/go have lock files though
https://docs.vllm.ai/en/v0.20.0/api/vllm/model_executor/laye...`vllm.model_executor.layers.quantization.turboquant`
> The technique implemented here consists of the scalar case of the HIGGS quantization method (Malinovskii et al., "Pushing the Limits of Large Language Model Quantization via the Linearity Theorem", NAACL 2025; preprint arXiv:2411.17525): rotation + optimized grid + optional re-normalization, applied to KV cache compression. A first application of this approach to KV-cache compression is in "Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models" (Shutova et al., ICML 2025; preprint arXiv:2501.19392). Both these references pre-date the TurboQuant paper (Zandieh et al., ICLR 2026).
These are very different media types with very different goals.
Is there a way to disable it? Sometimes I value agent not having knowledge that it needs to cut corners
FYI you can run just `uvx pdm`
Also got it, found this thread by googling "ycombiinator"
Also got it, found this thread by googling "ycombiinator"
Why? They acquired books, that’s what they do
Good rationalism includes empiricism though
llama.cpp is a mess and ollama is right to move on from it
Of course it is. GPT-5 is one of the most anticipated things in AI right now. To live up to the hype, it needs to be a reasoning model.
It’s unfortunate that llama.cpp’s code is a mess. It’s impossible to make any meaningful contributions to it.