A useful analogy is Prolog, which you can think of as a logical programming language. It has formal, logical rules as primitives, and the deductive inference search happens under the hood. Similarly, probabalistic programming languages have random variables and probability distributions as primitives and do probabalistic inference (usuallly Baysian) under the hood.
Say you have a model, y = mx + b, and you have some data. Normally you can solve this explicitly with matrices or whatever. But you'll need to implement the solution. With probabilistic language like BUGS, you can feed in the model and the data and it would return `m` and `b` as a probability distribution.
This is an trivial example. One example of where it really comes in handy is doing bayesian inference.