"...the backwards case ... basically comes down to performing approximate eigenvalue decomposition..."
Hmm, not in general. (I've been working on various problem-specific versions of the backwards case for most of my career.)
FYI, some folks (see: https://spectrallearning.github.io/icml2014/) are certainly working hard to apply eigenvalue decompositions (well, more like generalized SVDs) to any inference problems they can get their convexity-loving hands on.
Disclaimer: my lab mate has been organizing these workshops for the past couple of years and I'm currently procrastinating from preparing a presentation on this material for a reading group later this week, so I've presently got mixed feelings towards it.
- Start with a value + divergence and then propagate that state through your transforms to get a mean + divergent final state.
I should note that in the given example the base divergences need to be found through some method, which is usually running a Monte-carlo around some baseline state, and then making the significant assumption that new calculations (with small deltas) follow roughly the same statistical behavior.
For the backwards case, that can be a significantly more challenging case, as it basically comes down to performing approximate eigenvalue decomposition. However, there are a lot of SOTA tools like pretty much anything from this list:
- http://en.wikipedia.org/wiki/Pattern_recognition#Algorithms
IE, the crater example is fairly close to the setup for a Hidden Markov Model, where we get final data, and assume that it is based on a set of hidden states. Or for lighter reading, almost any of the major tools used in the Netflix recommendation contest, which is effectively an enormous version of this problem.