twiecki

There is a collection of curated resources here: https://www.pymc.io/projects/docs/en/stable/learn.html

If you are looking for an explanation of MCMC that focuses on intuitive understanding to complement more mathematical introductions, I wrote a blog post trying to explain things in simple terms here: https://twiecki.io/blog/2015/11/10/mcmc-sampling/

No, back-propagation would not give full Bayesian inference (although there are some tricks [0]). They instead use variational inference[1], which allows for fast inference of continuous PGMs.

[0] http://mlg.eng.cam.ac.uk/yarin/blog_2248.html

[1] https://arxiv.org/abs/1603.00788

Brian Moriarty gave a great talk in 2015 about his story of Loom with many interesting behind-the-scenes infos: https://www.youtube.com/watch?v=z1aVDael-KM Towards the end he also says he'd be excited to do a Loom sequel with the right studio.

There are packages that are just Python libraries, like PyMC3: http://pymc-devs.github.io/pymc3/ It allows you to specify a Probabilistic Program with a few lines of Python code (x ~ Normal(0, 1) becomes x = pymc3.Normal('x', 0, 1)) and then run automatic inference on whatever model you dream up, it's very powerful. I always found the approach of inventing a new language to specify Probabilistic Programs like Stan does a bit odd. There, you specify the model as a string and can't interact with it or debug if something goes wrong. Disclaimer: I'm a PyMC3 developer.

I don't think it has to be either or. For example, you can implement a Deep Net in a probabilistic programming framework (http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learn...) like PyMC3. The inference here is not using sampling but rather ADVI (http://pymc-devs.github.io/pymc3/api.html#advi) which is almost as general but much much faster, and can be run on sub-samples of the data (similar to stochastic gradient descent used in deep learning). Once we bridge these two domains we can get the best of both worlds, like a deep net HMM.

PyMC3 (http://pymc-devs.github.io/pymc3/) has all the powerful samplers that Stan has (i.e. NUTS) as well as support for discrete priors. It allows you to specify your models in pure Python, supports matrix algebra, and also has variational inference which allows for large-ish scale machine learning (e.g. here is a blog post where I train a Bayesian Neural Net on MNIST http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learn...). Under the hood, it uses theano for JIT compilation to C or GPU.

I'm one of the core devs.

Thanks for your comment! I did split the data in two, but it's easy to miss. X_test and X_train are the two sets. ann_input.set_value(X_test) then switches in the test values. That's identical to running make_moons() again.

If you're interested: http://pymc-devs.github.io/pymc3/

PyMC3 uses Theano to create a compute graph of the model which then gets compiled to C. Moreover, it gives us the gradient for free so that HMC and NUTS can be used which work models of high complexity.

I use it in production, despite it still being beta. We're close to the first stable release but there are still some small kinks to figure out.

Disclaimer: I'm a co-developer.

Anyone else notice that the code wasn't PEP8 compatible? ;)

I went ahead and fixed it: https://gist.github.com/twiecki/7361de09b8e4c0a6248e

The book is based on PyMC2, a Python module for Bayesian inference: http://pymc-devs.github.io/pymc/

PyMC3 is nearing the first beta release and is a complete rewrite that's using Theano to add just-in-time-compilation as well as support for next-generation sampling methods like Hamiltonian Monte Carlo (e.g. NUTS): http://pymc-devs.github.io/pymc3/

There is also the blue card (http://www.bluecard-eu.de/) which is pretty easy and quick to get if you have a university degree and a job-offer.