There was a paper from the Allen institute from around 2017 successfully using 1 bit quantization but I can’t find it right now. We started using it where I was working at the time but I’m no longer there so I don’t know how it all turned out.

Functional? Sure, it will work. Useful? Unlikely. It would be surprising to see quantization of pretrained models to surpass the existing research around binary NNs (trained from scratch as 1-bit while using full precision for the most critical parts)

If we get 1-bit quantization, wouldn't it be basically a bunch of nested if's and else's?

So, just like the rest of my code then

So a decision tree? They are good for some tasks I reckon

No. The catch is that what's quantized here is not individual weights but groups of them.

Those networks are named as discrete neural networks. There are already research on those, mainly for (homomorphic) encryption purposes (because it’s much easier to homomorphically encrypt NNs of just 0’s and 1’s than normal NNs).

I predict that even less-than-1-bit viable quantization methods will be found eventually. Of course, the bits-per-weight figure should be interpreted as something average: e.g., group-quantizing 32 weights into 24 bits would be 0.75 bits per weight.

Yes :) https://arxiv.org/abs/2310.16795