News for hackers. Stuff that matters.
Re:Fastest Autograd in the West by 317070 on Thursday January 04, 2024 @06:00PM (#38873501)
> 1. we test many computation graphs (graph is changing constantly) 2. many-many scalar operations with roughly 10k—100k nodes in each graph 3. every graph should be compiled and ran around 10k times both forward and backward

Now I'm curious to know where that is useful. Some kind of meta-learning approach?

Re:Fastest Autograd in the West by dimatura on Thursday January 04, 2024 @08:32PM (#38874621)
Automatic differentiation isn't just for machine learning, it's useful in several other applications where you have an optimization problem built of many variables interacting in differentiable ways. One possible guess could be structure-from- motion or SLAM, where the graph is often dynamic and several thousands of nodes isn't uncommon (the nodes would be things such as camera poses and 3D landmarks for example). However, in that case there are other frameworks that are built with that scenario in mind (Ceres, GTSAM), and would probably be better baselines.
Re:Fastest Autograd in the West by charcircuit on Friday January 05, 2024 @01:31AM (#38876376)
Optimizing something with AD is machine learning.
Re:Fastest Autograd in the West by kragen on Friday January 05, 2024 @02:33AM (#38876664)
i want to point out that in many cases you can compute a differentiable relaxation of a nondifferentiable problem to get something you can optimize more easily (with, for example, automatic differentiation) and then use the relaxation to solve the original nondifferentiable problem, for example with branch-and-bound
Re:Fastest Autograd in the West by HarHarVeryFunny on Thursday January 04, 2024 @08:00PM (#38874398)
Based on the "we TEST" wording, maybe evaluating graphs for fitness - some sort of genetic/evolutionary approach perhaps?
Re:Fastest Autograd in the West by mratsim on Thursday January 04, 2024 @06:05PM (#38873561)
Maybe instruction-level autograd. When you use a differentiable programming language.
Re:Fastest Autograd in the West by arogozhnikov on Friday January 05, 2024 @02:20AM (#38876612)
Robotic planning of liquid handling. There are many potential alternative ways to achieve the same result, each resulting in its own computation graph.

I imagine any kind of trajectory planning for many agents faces similar challenges (e.g. robots in a factory).

Re:Fastest Autograd in the West by 317070 on Friday January 05, 2024 @04:40AM (#38877297)
But these computation graphs are similar every time. You would/should write these as a small graph with conditionals on large tensors, not as a fast changing graph of scalars. The latter will always be slower when you have an accelerator like a GPU.
Re:Fastest Autograd in the West by arogozhnikov on Friday January 05, 2024 @05:51PM (#38886146)
> But these computation graphs are similar every time.

For now I'm testing one-after-another because failure of previous (overflow, too-small-transfer, etc) gives a hint about which part can be changed in the graph.

I've been thinking about vectorizing over multiple guesses at once, but don't see a fast reliable way to merge several graphs (also that induces significant burden on other parts of the project - autograd is just one component).