Correct me if I'm wrong, conceptually, probabilistic programming does to equation solvers what declarative language like SQL do to query data.

Say you have a model, y = mx + b, and you have some data. Normally you can solve this explicitly with matrices or whatever. But you'll need to implement the solution. With probabilistic language like BUGS, you can feed in the model and the data and it would return `m` and `b` as a probability distribution.

This is an trivial example. One example of where it really comes in handy is doing bayesian inference.

A useful analogy is Prolog, which you can think of as a logical programming language. It has formal, logical rules as primitives, and the deductive inference search happens under the hood. Similarly, probabalistic programming languages have random variables and probability distributions as primitives and do probabalistic inference (usuallly Baysian) under the hood.

Probabilistic programming is not really "programming" as we might use the term without qualification. It's wholly focused on the specification of probabilistic models, their structure, data, and evaluation pattern.

Consider BUGS as it's a very prototypical probabilistic programming example. If your goal is to specify that you have a model where you observe the "wetness of the grass outside" and infer whether it "rained previously" then you can use BUGS to express it easily.

    model {
      rained ~ dbinom(0.1) /* prior probability */
      for (i in 1:N) {
        grass[i] ~ rained
      }
    }

for some data grass[i].

(That was a very sketchy example, don't read into it too closely)

The heart of this is that quantitative methods are getting so well understood that we can express the algebra of models as a formal language. When we do this we get many of the benefits that programming has done for formal expressions of flow-charts.

While the line isn't sharp, I would distinguish somewhat between older BUGS like languages, where the underlying analogy is typically a directed graph, and more recent probabilistic programming languages like Church, where the underlying analogy is an execution trace. The excitement around this newer form of probabilistic programming comes from the fact that they really are programming languages. Unlike BUGS, they inherit all the flexibility, composability, and encapsulation of the underlying language. Inference is viewed as sampling execution traces, so the connection with a programming language is quite tight.

I agree in practice, but pedantically, blurring that line is the second step to understanding modern PPLs.

A probabilistic programming language is probably better understood as a way to model stochastic processes. What I mean by modeling stochastic processes is we have data that we know was generated by some non-deterministic process. We also don't know all the parameters of this process. Probabilistic programming languages make the point that the best way to specify these processes is as programs. Once we do that, we can in a sense run these simulations in reverse to infer parameters in our models and even infer values for missing data.

It depends on your model - size and variables. If you have a moderately sized model with discrete variables you can infer values using exact algorithms (e.g. clique tree propagation, variable elimination, etc). If your model is too complex or has certain continuous distributions then you can use simulation techniques or other approximate algorithms.

Given that probabilistic graphical models is in general NP-hard (or #P-hard) then approximate algorithms are often used. However, for many problems discrete valued networks can be effectively managed with exact algorithms.

Some tools for exact inference (that are free or commercial with free versions) are:

http://genie.sis.pitt.edu http://www.norsys.com/netica.html http://www.hugin.com

Netica in particular has a large library of example networks.

That doesn't really change my definitions. I agree exact inference is intractable for anything but modest problem sizes. Luckily, I think most work these days is in making these systems use approximate inference.