Re:Thermodynamic Natural Gradient Descent
by thomasahle on Friday May 24, 2024 @02:12PM
(#40468898)
The main point of this is that natural gradient descent is a second-order method. The main GD update equation is:
∇̃L(θ) = F⁻¹∇L(θ)
which requires solving a linear system. For this, you can use the methods from the author's previous paper [Thermodynamic Linear Algebra](https://arxiv.org/abs/2308.05660).
Since it's hard to implement a full neural network on a thermodynamic computer, the paper suggests running one in parallel to a normal GPU. The GPU computes F and ∇L(θ), but offloads the linear system to the thermo computer, which runs in parallel to the digital system (Figure 1).
It is important to note that the "Runtime vs Accuracy" plot in Figure 3 uses a "timing model" for the TNGD algorithm, since the computer necessary to run the algorithm still doesn't exist.
∇̃L(θ) = F⁻¹∇L(θ)
which requires solving a linear system. For this, you can use the methods from the author's previous paper [Thermodynamic Linear Algebra](https://arxiv.org/abs/2308.05660).
Since it's hard to implement a full neural network on a thermodynamic computer, the paper suggests running one in parallel to a normal GPU. The GPU computes F and ∇L(θ), but offloads the linear system to the thermo computer, which runs in parallel to the digital system (Figure 1).
It is important to note that the "Runtime vs Accuracy" plot in Figure 3 uses a "timing model" for the TNGD algorithm, since the computer necessary to run the algorithm still doesn't exist.