Capacitors (well, RC systems) are exponential in time, but their relationship between voltage and current is purely linear. So if you want to use capacitors to do exponentials, and your machine inputs are in voltage or current, you need a way to convert a voltage or current to a time delay.
More typically, I think electronic analog computers used the exponential region of a semiconductor diode to do logarithms and exponents, and thus multiplication and division, in nanoseconds, using 1950s technology. You can use capacitors and inductors to integrate and differentiate time-varying functions.
It's also pretty neat you can get an approximate solution in a single step, rather than cranking through a bunch of operations. It seems like a compression problem in information theory, especially when you consider the two step or multi step graphs.