Maybe we should make "Brain Damage Factor" a official term to denote how much types of quantizations degrade output compared to unquantized.:)

I think that's known as degradation, but I think the brain damage metric could be usefully applies to the poor sods who try to get the quantization to work in the first place.

Lol :)

There's actually an old paper titled Optimal Brain Damage, where they don't try to find optimal quantizations, but optimal sparse versions of a models-- i.e. where some weights are set to zero.

Interesting, I heard something like that, but now I must read about it.

I found…

Optimal Brain Damage https://www.researchgate.net/publication/221618539_Optimal_B...

Optimal Brain Compression https://openreview.net/pdf?id=ksVGCOlOEba#:~:text=The%20resu....

TinyVolt’s implementation of it: https://github.com/TinyVolt/optimal-brain-compression

Thanks, I'll read it up. Interesting.

That was really for them. You’re out there building neat stuff. Your talent might warrant looking into AdderNets and Bitnets which might get the cost down. There’s also some brain-inspired designs.

I don’t think many people have implemented such things. You might discover something new experimenting with them.

:)