https://en.wikipedia.org/wiki/Logarithmic_number_system this has been tried before. The hard part is getting addition to work fast enough.

Would addition be slower than for normal floats?

much. To add 2 lns numbers, you need to essentially compute ln(1+b). There isn't a good algorithm to do so (except for lookup tables for low precision)

Silly idea: what if we ran ML models on pure logarithms and used some completely terrible approximation for addition (maybe just the max function, or max plus a lookup table based on the difference between the two numbers when it's small)

once you're down to 8 bits or so you can totally do this with a lookup tables (for 8 bits, you need 512 bytes of table which isn't too bad.

It's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3]

[0] https://arxiv.org/abs/1603.01025

[1] https://par.nsf.gov/servlets/purl/10163070

[2] https://arxiv.org/abs/2310.11453

[3] https://arxiv.org/abs/1606.01981

Variants periodically show up on HN. https://news.ycombinator.com/item?id=43181610