News for hackers. Stuff that matters.
Re:Higher Order Derivatives of Transforms by magicalhippo on Monday January 29, 2024 @09:50AM (#39176866)
The previous post in the series[1], helpfully linked to in the introduction, has a lot more details of the underlying concepts.

It also set my mind wandering to the not-technically-related functional derivatives[2], where you vary the function slightly rather than the argument value.

I'm not great at math, but I do love this what-if exploration you can do in math. Due to the various proofs underlying it all it seems sometimes more fruitful than similar exploration in programming, where one might quickly stumble upon obscure compiler errors or similar obstacles.

[1]: https://nosferalatu.com/DerivativesLogarithmsTransforms.html

[2]: https://en.wikipedia.org/wiki/Functional_derivative

Re:Higher Order Derivatives of Transforms by abhgh on Monday January 29, 2024 @05:41PM (#39183783)
If you're looking for interesting derivative-adjacent ideas, I would also recommend Clarke derivatives [1]. They occasionally show up in ML papers, e.g., [2], [3]. Unrelated bu tangential, another place where you need derivatives but don't have access to them (standard or otherwise) is in the area of black-box optimization. Within this area, Bayesian Optimization (BayesOpt) has picked up quite a bit, which I've successfully used quite a bit in my work - I've an introduction here [4]. There is also a good book available online for free on the topic [5].

[1] https://en.wikipedia.org/wiki/Clarke_generalized_derivative

[2] https://proceedings.neurips.cc/paper/2021/file/70afbf2259b44...

[3] http://proceedings.mlr.press/v202/lee23p/lee23p.pdf

[4] https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs

[5] https://bayesoptbook.com/

Re:Higher Order Derivatives of Transforms by ogogmad on Monday January 29, 2024 @06:33PM (#39184256)
Functional derivative : Frechet derivative :: gradient : directional derivative