Re:Hot Chips 34: AMD’s Instinct MI200 Architecture
by p1esk on Sunday September 18, 2022 @11:40PM
(#32894030)
There will be no hw product able to run inference in purely analog domain in foreseeable future. There’s simply no good practical way to store analog signals yet. And unfortunately activations must be stored in models like transformers or convolutional networks.
We already have mixed signal accelerators (e.g. Mythic) which are not much faster than digital competition (for many reasons).
From this point of view there’s no difference between inference and training, especially as FP8 format is being adopted by Nvidia and others.
Re:Hot Chips 34: AMD’s Instinct MI200 Architecture
by Taniwha on Monday September 19, 2022 @01:11AM
(#32894503)
The big problem with analog is power - we have 40 years of building gates that are on or off and use virtually no power in those states because no current flows - the problem with analog is ohm's law you need to have current flowing to make analog work
For training, I’m certainly interested but not at all convinced that it will dominate. I work on multiple projects right now where the reduced dynamic range of analogue signals would be a complete non-starter given the problem domain. I’m not sure how they get around that.