Parallel tempering should also be exploited more.
That said, at the end of the day, the ideal sampler would be able to reason about the distribution as program and not just as a mathematical black box. It should build tractable approximations intelligently and use those to bootstrap exact sampling schemes.
I think we're going to see a wealth of better samplers come out in the next decade, following the path of combinatorial optimization towards preserving the structure of the programs.
aren't there methods for online sparse estimates of the Hessian?
I'd expect a lot of "large" problems for which RMC is useful would have sparse structures.
> ideal sampler would be able to reason about the distribution as program
Are you a developer of Stan? If not, you might be interested.
_edit_ by online estimate of a Hessian, I meant online numerical approximation based on the sequence of Jacobians.