Capacitors (well, RC systems) are exponential in time, but their relationship between voltage and current is purely linear. So if you want to use capacitors to do exponentials, and your machine inputs are in voltage or current, you need a way to convert a voltage or current to a time delay.

More typically, I think electronic analog computers used the exponential region of a semiconductor diode to do logarithms and exponents, and thus multiplication and division, in nanoseconds, using 1950s technology. You can use capacitors and inductors to integrate and differentiate time-varying functions.