> Analog computers making a comeback ? you never know.

For dataflow paradigms like deep neural nets it makes sense. If it's significantly faster and significantly more power efficient, and the intermediate states are largely arbitrary weights anyway, what's the point of digital precision.

It's not the precision, it's avoiding O(n^2) physical connections per layer.

That's my point. Swapping digital for analog gives up the ability to know the precise bits of a register but in return you gain a lot in efficiency.

That's not what I'm getting at. It's not the nodes you need to worry about, it's the edges. If you've got two fully connected 100-neuron layers in a physical circuit, that's 10,000 wires you need to route, and they all want to cross each other. That's why nobody builds physical neural networks.

I wonder if using this for low-precision floating point math in graphic cards would make sense.

Maybe it's a way to do realtime raytracing? Who cares if the results have a little noise? It can be even argued it gives it a special "look".

> Analog computers making a comeback ? you never know.

Analogue processing may be a superior alternative to GPGPU for implementing neural networks.

Where does the energy go in neural network workloads?

Wires.

To and from memory, in the memory dice, on the compute die, in the register files, etc.

Check out the paper "DNN Dataflow Choice is Overrated". And that's for the most compute bound cnns!

Edit: and if you think there's enough SNR to make analog "deep" processing in memory work, go make a quick billion dollars by expanding commodity memory with multi level. If you can add another level to flash you'll get rich quick.

I dispute this. The problem with analogue processing (besides noise, of course) is that the dynamic range sucks compared to plain old floating point. Fine for inference, not for training. And most of the compute power in NN is spent on training, inference is trivial.

Analogue processing has certainly been successful for people, so I wouldn't be too dismissive.

I would not be certain of the premise in your statement. We do not understand the brain well enough to say whether error correction (which is inherently digital) is utilized. There are some pretty solid clues that some of the processing in the brain is not analog (e.g. the step-like behavior of the activation function of biological neurons).

And coming from the other side: If it turns out that the brain is actually an analog computer without anything that can be rephrased as some form of sloppy noisy approximately-digital error correction, this would be incredibly important discovery, completely changing our understanding of theoretical computer science and complexity theory. P-vs-NP or the eventual creation of quantum computers would pale in comparison to the impact of discovering scalable analog computers.

Agree mostly, except for "error correction .. inherently digital"

For an example of an analogue error corrector, see https://en.wikipedia.org/wiki/Centrifugal_governor

PS. cool fact; "governor" has the same etymology as kubernetes.

Inference is typically much more power constrained, as it is often performed on a battery. For example in a mobile phone, or in an IoT sensor unit. This makes approximate computing for neural network accelerators an interesting field of research.

Nailed it: https://www.mythic-ai.com/technology/

Probably not analog computers. But analog circuitry, CPU, and power supply all in one package seems likely.

Analog computers are incredibly fast for solving some very difficult problems such as integration. I could see, if easy to implement, basically an "analog FPGA" hybrid of digital fabric and analog components for you to piece together.

You may be interested to know that FPAAs are a thing that does exist. https://www.anadigm.com/_doc/DS030100-U006.pdf

That's really cool, thanks for sharing!

They're also noise-sensitive, hard to calibrate, hard to reprogram, and only suitable for a fairly small range of problems. But nevertheless very cool devices.

Bloody awful things. I’ve built a few simple analogue PID loops and simple analogue computers over the years and it’s an art and a half. I think the biggest pain I had was before the availability of reasonably stable log converters was trying to use matched transistor pairs to perform log operations. Took three days to work out where an error was creeping in and it was from the thermal offset voltage of the buffer opamp input. The circuit was drifting due to my desk lamp warming it up. The IC was out of spec on drift too! :(

I did succeed in producing a four quadrant multiplier in the end but required some unholy things to be done.

That and noise and the cost of precision make it almost useless compared to a digital computer now. The rise of direct sampling SDRs confirms this. Crazy dynamic range, excellent noise floor and hardly any analogue up front.

The 4QM is one of the most useful analog devices in music synthesis, where absolute precision is unnecessary. There it's called a ring modulator because it emulates the sound of a bell. It produces both the sum and the difference of the two sinewaves fed into it.

Edit: Just looked up "ring modulator" on Wikipedia. It says the name comes from the ring of diodes in the canonical circuit. I'd always heard the genesis was from the bell-like sound it made. And there are better ways to build analog multipliers than four diodes in a circle.

Similar to diode balanced mixers in RF stuff.

Yep. It's a mixer in RF terminology. Or a heterodyne. Lots of names for this thing.

Try to find some analog device with 2\^53 signal to noise ratio, look how much it costs.

Then remember that this is only the ration on our most used digital numbers because it's easy and cheap. Applications that require precision normally go with a 2^113 ratio.

I thought some NNs actually used fewer bits (32?) than the normal 64bit floating point because it is faster and good enough?

On PCs everybody uses 64 bits, and if they choose not to, they have a large risk of being converted into and out of it all the time by the compiler because it's faster.

But specialized CPUs often get different number sizes. It's common for specialized NN and GPU hardware to use 32 bits floats, what leads to 2^24 definition that is actually around the top precision you will find for analog components on the places people (instead of labs) go for buying them, at least on the countries where access to those aren't restrict due to nuclear proliferation concerns.

Floats on modern PCs are 32bit (you can look that up if you want). Doubles (double precision floats) are 64bit and are seperated from 32bit. There is also half precision float (16bit) since quite a few generations.

There is no conversion happening on the compiler level unless you choose to run with soft-floats for reasons unknown.

You don't need specialized GPU hardware for 32bit floats either; that is the default. You need specialized hardware if you want 64bit floats however (CAD Designers love those). Half precision is also available on all GPUs that people will likely care about.

Some NNs use 16bit floats instead of the normal 32bit floats, because yeah, it's good enough to cover what you need from low quality images for example (you get around 12bit color space, which can be enough for a lot of things)

I guess all you would have to do is to make sure the precision was a few dB/bits more accurate than 16/32/64 bit floating point for a certain sigma probability and a lot of people would be perfectly happy with it.

I wonder if you could use it as an efficient starting point for various different integer division algorithms which rely on a rough starting point. For instance, it could be your X_0 estimate for (1/D) in the Newton-Rhapson method [1]. Or, potentially, as a high-quality first stage input for a successive-approximation approach [2].

This could allow you to get a rough approximation incredibly quickly and opt to refine it further using it as a high-quality input to speed up finding the "true" answer.

[1] https://en.wikipedia.org/wiki/Division_algorithm

[2] http://www.ijiee.org/papers/173-E0005.pdf

The trade off would be that your A-D converter would have to be fast enough as well as offer enough bits of result to give the iterative algorithm a running start. Digital video cameras would be the class of device we ~all have that does high-samples-per-second conversion. A 4K@60p camera has to do 500Ms/s at least so if you can allow that many approximate results per second then it might work out. There would also have to be a comparably fast DAC to tell the analog circuit what you want to approximate.

I think a more-complete computing engine in analog would be a bigger win, because there won't be conversion overheads. Or as was suggested somewhere in this topic, a field-programmable analog array.

>Analog computers making a comeback ? you never know.

The main advantage they're advertising for the analog parts is that you don't need an ADC or DAC. That's not helpful for computers, but it can be very helpful for specialized signal processing parts.