A useful analogy is Prolog, which you can think of as a logical programming language. It has formal, logical rules as primitives, and the deductive inference search happens under the hood. Similarly, probabalistic programming languages have random variables and probability distributions as primitives and do probabalistic inference (usuallly Baysian) under the hood.