You may be interested to know that FPAAs are a thing that does exist. https://www.anadigm.com/_doc/DS030100-U006.pdf

That's really cool, thanks for sharing!

They're also noise-sensitive, hard to calibrate, hard to reprogram, and only suitable for a fairly small range of problems. But nevertheless very cool devices.

Bloody awful things. I’ve built a few simple analogue PID loops and simple analogue computers over the years and it’s an art and a half. I think the biggest pain I had was before the availability of reasonably stable log converters was trying to use matched transistor pairs to perform log operations. Took three days to work out where an error was creeping in and it was from the thermal offset voltage of the buffer opamp input. The circuit was drifting due to my desk lamp warming it up. The IC was out of spec on drift too! :(

I did succeed in producing a four quadrant multiplier in the end but required some unholy things to be done.

That and noise and the cost of precision make it almost useless compared to a digital computer now. The rise of direct sampling SDRs confirms this. Crazy dynamic range, excellent noise floor and hardly any analogue up front.

The 4QM is one of the most useful analog devices in music synthesis, where absolute precision is unnecessary. There it's called a ring modulator because it emulates the sound of a bell. It produces both the sum and the difference of the two sinewaves fed into it.

Edit: Just looked up "ring modulator" on Wikipedia. It says the name comes from the ring of diodes in the canonical circuit. I'd always heard the genesis was from the bell-like sound it made. And there are better ways to build analog multipliers than four diodes in a circle.

Similar to diode balanced mixers in RF stuff.

Yep. It's a mixer in RF terminology. Or a heterodyne. Lots of names for this thing.

Try to find some analog device with 2\^53 signal to noise ratio, look how much it costs.

Then remember that this is only the ration on our most used digital numbers because it's easy and cheap. Applications that require precision normally go with a 2^113 ratio.

I thought some NNs actually used fewer bits (32?) than the normal 64bit floating point because it is faster and good enough?

On PCs everybody uses 64 bits, and if they choose not to, they have a large risk of being converted into and out of it all the time by the compiler because it's faster.

But specialized CPUs often get different number sizes. It's common for specialized NN and GPU hardware to use 32 bits floats, what leads to 2^24 definition that is actually around the top precision you will find for analog components on the places people (instead of labs) go for buying them, at least on the countries where access to those aren't restrict due to nuclear proliferation concerns.

Floats on modern PCs are 32bit (you can look that up if you want). Doubles (double precision floats) are 64bit and are seperated from 32bit. There is also half precision float (16bit) since quite a few generations.

There is no conversion happening on the compiler level unless you choose to run with soft-floats for reasons unknown.

You don't need specialized GPU hardware for 32bit floats either; that is the default. You need specialized hardware if you want 64bit floats however (CAD Designers love those). Half precision is also available on all GPUs that people will likely care about.

Some NNs use 16bit floats instead of the normal 32bit floats, because yeah, it's good enough to cover what you need from low quality images for example (you get around 12bit color space, which can be enough for a lot of things)