Re:Fastest Autograd in the West
by 317070 on Friday January 05, 2024 @04:40AM
(#38877297)
But these computation graphs are similar every time. You would/should write these as a small graph with conditionals on large tensors, not as a fast changing graph of scalars. The latter will always be slower when you have an accelerator like a GPU.
[ Parent
| on HN ]
Re:Fastest Autograd in the West
by arogozhnikov on Friday January 05, 2024 @05:51PM
(#38886146)
> But these computation graphs are similar every time.
For now I'm testing one-after-another because failure of previous (overflow, too-small-transfer, etc) gives a hint about which part can be changed in the graph.
I've been thinking about vectorizing over multiple guesses at once, but don't see a fast reliable way to merge several graphs (also that induces significant burden on other parts of the project - autograd is just one component).