kadushka
kadushka on Hacker News
They use a meaningful portion of compute.
Just as $500 makes no sense nowWhy? You can use in codex, right?
We are being productive here!
This makes perfect sense. There are no real improvements anymore (just benchmaxxing), and they explain it by "pacing the frontier".
Guess why I'm posting on HN right now instead of working?because fable/astra is working for you? :)
CMU is #2 in the country for CS program. If those students don't think they can find a job, it's bleak.
the average information content of transformer LLMs at about 3-4 bits per parameterThe problem is that 4-bit block-wise quantization does not guarantee preserving 4 bits of useful information per parameter - not even on average. It simply assigns one of 16 quantization levels to each weight, with the whole block sharing the same scale/range.
How efficiently those 16 levels preserve the model’s information depends on the weight distribution, block size, range/clipping strategy, outliers, and which weights are actually important. Some weights may be represented almost exactly, while others lose much of their useful information.
A simple example is an outlier: if you choose the range to preserve a very large weight, much of the 16-level dynamic range is spent on that outlier, leaving coarse resolution for all the smaller weights in the block. So 4 bits of storage does not imply 4 bits of useful information preserved. Yes, QAT helps, but usually at the cost of learning efficiency. It takes longer to train a model to the same quality when using less precision, and sometimes we simply cannot get to the same quality level with not enough precision in the right places.
Another problem in quantization is that we don't really know which weights are sensitive - we can compute various sensitivity metrics, and some of these metrics will correlate with accuracy on some benchmarks, but not on others.
Another complementary option is, if the model is fast enough, we should be able to push up correctness by self-consistency voting at close to T=1. Smart/fast Zero-shot classifiers like the recent Jev could help with aggregation across answers too, extending applicability.
I'm not convinced by this argument - if such a method improves accuracy of a degraded quantized model, then it could in theory also help non-degraded full precision model. And if so, then we are back to square one, because this composite model will then get degraded due to quantization (baseline has improved!)
We do know one thing - increasing the size of the model usually makes it more robust to quantization. If going from 8 bits to 2 bits speeds things up by a factor of, say, 4x, then if we double the size of the model, we might still end up with an overall speedup. Finding this balance might become a hot area of research.
That's what I meant - we are currently use fp4 formats for training, and we cannot quite get away with that, despite dynamic quant and small block size - we still have to use quite a bit of higher precision (fp8 or even fp16) in various model components.
By “work out” you mean no accuracy degradation? That’s a big ask - currently we can barely quantize to dynamic fp4 with small block size - still not completely lossless on all benchmarks.
Everyone I know loves AI. It is actively making their lives better. All kinds of people, all kind of ages - universally love it and use it every day for all kind of stuff (shopping, health, learning, home improvement, etc, etc).
The article is well-written, and is very clear, thank you!
Cost is absolutely an issue here - my time is worth approximately $1000/day, so if a slightly worse model wastes one more hour of my time a day than the best model, it costs the company >$2k/mo. Fortunately my employer understands this well and encourages me to use the best models as much as I can.
Opus 5 and 5.6 Sol are definitely not smart enough to do my job. They require constant supervision. So why would I want to switch to even worse model? Even if it's just slightly worse?
How would this work? Where do the money for their pension fund come from? Would taking money from it result in them receiving smaller pensions?
Most quant papers I've seen usually report non-trivial degradation on standard benchmarks, like 1-10% degradation (compared to FP16/BF16). Especially when using 4 bits or lower. For example, I just opened a random paper:
https://arxiv.org/pdf/2410.09426 see Table 1.
p.s. dense vs MoE: both are being released because they offer different trade-offs: at the same level of quality, MoE will use less compute, but more memory.
models are getting weirdly good at hacking while still sort of sucking at a bunch of economically valuable taskslike most human hackers
taser/pepper spray within 5 years, firearms within 10.
We are surrounded by black boxes we depend on - have been for at least a century.
Probably had a lot of meetings
1. If you have good results on sufficiently large models (check latest papers re: which benchmarks are still relevant), post them on Github, along will detailed instructions how to reproduce.
2. Post the link to the GH repo in "Show HN" section.
3. If results are solid, write up a paper and upload it to arxiv. Next step would be try to publish in an ML conference.
p.s. To increase your chances of anyone actually clicking on your GH link, use good old Pytorch.
I'm not interested in gaming, but if you had a version for AI, I'd be using it!
Diffusion models are not autoregressive but have the same limitations
Imagine that we made an LLM out of all dolphin songs ever recorded, would such LLM ever reach human level intelligence?It could potentially reach super-dolphin level intelligence
I'm an employee, and my boss loves me because I deliver things he wants quickly and reliably - because I use AI tools. Guess who he will keep in the next round of layoffs?
I'm serious - the productivity boost I'm getting from using AI models is so significant, that it's absolutely worth paying even 2k/month. It saves me a lot of time, and makes me deliver new features much faster (making me look better for my employer) - both of which would justify spending a small fraction of my own money. I don't have to, because my employer pays for it, but as I said, if I had to, I would pay.
I would probably pay $2000 a month if I had to - it's a small fraction of my salary, and the productivity boost is worth it.
Sure, could be just lucky. But if there are several successful small studies, and several unsuccessful large ones (no idea if this is the case here), we should probably look for a better explanation.
use a diverse populationIf that's the case, we should question whether different homogeneous population groups respond differently to the substance under test. After all, we don't want to know the "average temperature of patients in a hospital", do we?