Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

Re:Show HN: Boldly go where Gradient Descent has never gone ... by zaitanz on Sunday May 26, 2024 @06:06PM (#40485826)

So this confuses me slightly and I am keen to know the advantage of using this. I work on projects that heavily use auto-differentiation for complex models. The models are defined by user input files at run-time, so the state and execution pathway of the model is unknown at compilation time. Would this help?

Given that all auto-differentiation is an approximation anyways. I've found existing tooling (CppAD, ADMB, ADOL-C, Template Model Builder (TMB)) work fine. You don't need to come up with a differentiable problem or re-parameterize. Why would I pick this over any of those?

[ Parent | on HN ]

Re:Show HN: Boldly go where Gradient Descent has never gone ... by big-chungus4 on Monday May 27, 2024 @08:39AM (#40490206)

In `if i > x`, derivative with respect to x is mathematically 0 at all points. DiscoGrad gives you a useful smooth approximation that is not 0 and lets the function learn those conditional values.

[ Parent | on HN ]