News for hackers. Stuff that matters.
Re:Why Probabilistic Programming Matters by tlarkworthy on Tuesday July 15, 2014 @09:24PM (#8040116)
I disagree. People who know how to build complex bayesian models will be over the moon they don't have to piece a complete system together using matrices. I see no reason for such a system not to scale, essentially the underlying math is elementary, but each statistical package at the moment has a kind of lock in though lack of interoperability. Ideally you want to only do MCMC as a last resort for part of the reasoning, but once you are in WIN bugs you end up doing all the basic stuff with MCMC too, as its too much of a pain to try and tie it to a more efficient system for the bits you can reason with efficiently. I see potential in models that are partially MCMC and partially analytical. I see even more potential in embedding those kinds of systems within larger smart systems.

Some probabilistic standardisation would be lapped up by the applied community.

Re:Why Probabilistic Programming Matters by nashequilibrium on Tuesday July 15, 2014 @09:57PM (#8040202)
I am enjoying going through this "Probabilistic Models of Cognition":https://probmods.org/generative-models.html .Even though i am a python guy but the fact that it is written using a functional probabilistic programming language called Church, really makes it easy to follow along.
Re:Why Probabilistic Programming Matters by tlarkworthy on Tuesday July 15, 2014 @10:19PM (#8040261)
yeah looks like nice externals, but yet again, all the inference is with MCMC :s

MCMC is the most general approach, but also the most inefficient, so its got limited appeal in crunching live numbers coming out a physical system. Some problems are intractable and its the only way, but really you want to limit application of MCMC as little as possible, so if you use church you now have an interoperability problem with binding (LISP) to some ugly but efficient FORTRAN system or some such.

Exactly why I think think this initiative will be highly welcome!

Re:Why Probabilistic Programming Matters by mjw on Wednesday July 16, 2014 @06:35PM (#8045005)
So you're probably aware of this, but I think it might be interesting to try and articulate why I fear extending fully-Bayesian inference algorithms beyond MCMC could be a challenge, from experience with ML-focused Bayesian models and larger datasets:

Generally to make inference for these kinds of models fast, you have to make it approximate.

MCMC has the nice property that, while it's an approximate method, it becomes exact in the limit of infinite samples. Meaning that when applying it in a general setting, the compiler doesn't have to make hard and final decisions about where and how to introduce approximations -- it produces MCMC code and you decide how long you can be bothered to wait for it to converge and how accurate you want the answer to be, based on various diagnostics.

Most other non-MCMC-based approximate inference methods (MAP, EM, Variational Bayes, EP, various hybrids of these and MCMC...) don't converge to the exact answer, they converge to an approximate answer which remains inexact no matter how long you run the algorithm for.

Different approximations have different strengths and weaknesses, and the best choice may depend on the model, the data, what you actually want to do with the posterior (a mode, a mean, an expected loss, predictive means, extreme quantiles?), and what frequentist properties you want for the results of the inference, especially given that this is no longer pure Bayesian inference but some messy approximation to it. Often the only practical way to decide will be to try a bunch of different things and see which does best on the application (or best predicts held-out data), even armed with full human intuition. A compiler doesn't really have a chance here.

In short, it's not going to be easily to automate fully because it's not something you can decide on formal grounds alone. You have to decide how and where you're willing to approximate, and you have to understand the approximations used and check the resulting approximated posterior against data in order to know whether to trust the results and how to interpret them.

Re:Why Probabilistic Programming Matters by mjw on Wednesday July 16, 2014 @03:29PM (#8044002)
I don't think I'm disagreeing with you really. I'd love to have a standard language for describing probabilistic models, together with some tools to automate deriving a range of different approximate inference algorithms for them. Perhaps also some heuristics on top of this to pick good defaults for the inference method, which I can override. Ideally also a way to implement my own approximations (more realistically, approximations from papers I want to implement) without having to start from scratch.

I'd prefer to decouple those things though, so I can benefit from the first and the second without waiting for a "sufficiently smart compiler" to arrive which can automate the whole shebang well enough to rely on in a fully general programming language setting. That was what I was expressing skepticism about, although I'd be very happy to be proven wrong.