Re:Tesla Project Dojo Overview
by krasin on Monday August 23, 2021 @02:01AM
(#28272714)
ML training is a latency sensitive problem that benefits from locality: most of the learning algorithms wait for all parallel workers to send gradients back before updating weights. While workarounds exist, life is easier without them.
[ Parent
| on HN ]
Why is this a necessary constraint? (Not trolling, just curious if this has been investigated.)
Our neurons don't have a global clock, it doesn't seem to be a problem for us. My intuition is that as long as the input is changing continuously and not through random presentation, and at a rate where big changes happen at a lower order of magnitude than the average compute rate, it wouldn't matter all that much in terms of accuracy.