When I took statistics in college, we started with a rather basic definition, something to the effect of: "A statistic is a function performed on a set." Statistics studies what you can infer when you know something about a set, but not everything about it, namely its precise contents. Often, what you are told about a set is something about its probability distribution, thus linking probability and statistics together.

A useful parallel can be drawn with situations involving measurements and data, since data often have the same feature of telling us something but not everything. This is what I believe makes statistics so useful for science.