Conservation of angular momentum. Stars and their planetary systems form out of clouds of dust and gas which collapse into a fuzzy disk and then into bodies. The sum of the angular momentum before and after has to be the same so there is a big statistical pressure towards all bodies rotation and orbital planes to be near some average overall plane, there are gravitational forces at work too. Orbits over poles are as far off plane as possible and so are least likely.

Then why don't they orbit in the same direction, as the planets do? Aren't many moons gravitationally captured in any case?

Gravitational capture is the only way retrograde orbits could happen - a blob of spinning dust and gas just won't produce moons orbiting in opposite directions. Anything going in the "wrong" direction will be subject to a whole lot of drag forces and lose all of its orbital speed and fall inwards. Retrograde orbits have to happen by grabbing something late or after the formation phase.

All of the discovered retrograde orbiting moons are in similar orbits which suggests they all came from some larger object broken apart. Keep in mind all of these new objects are quite small, just a few miles across. About a billion of them could fit inside Saturn's largest moon, Titan (seventeen were discovered).

Regular moon fits with what I was talking about before.

Irregular moons are the ones with eccentric, retrograde, or otherwise weird orbits and acquisition histories.

The more you get into specifics about this kind of thing, the less concise and accurate you can be, it would take an expert up to date with the literature to go much deeper.

>Keep in mind all of these new objects are quite small, just a few miles across. About a billion of them could fit inside Saturn's largest moon, Titan (seventeen were discovered).

With planets, we started getting too many "planets" when we discovered more and more really tiny ones, so we started coming up with new names for them: "dwarf planet", "Trojan", "asteroid", etc. A small rock in the asteroid belt orbits the Sun, but it still isn't called a "planet".

It sounds like we need to do something similar with moons, or else it's going to get out of control. There's likely an absurd number of small rocks orbiting Saturn and Jupiter; should they really all be called "moons"?

"should they really all be called "moons"?"

Well, some of them are called "rings"...

Moons generally orbit on the same plane as their planet's orbit plane

Is this because polar orbits are unstable in some sense, or because pretty much everything already is in the planet's plane?

That's a good question, and I don't think you'll find an especially convincing answer.

Polar orbits are "unstable" in a few senses, namely that even in an ideal setting orbital planes will precess around central masses that are lumpy rather than spherically symmetric; rotation of a lumpy central mass introduces further precessions. Nodal precession is pretty interesting, for example. One can concoct all sorts of odd orbital changes by perturbing the rotating central mass "under" a test object in stable orbit.

However, in the case of known solar system bodies, slowly shifting a satellite in near-polar orbit into a near-equatorial orbit conflicts with quite a lot of evidence; by the mid 1960s it was pretty safe to say that known inner moons maintain over very long timescales an almost constant orbital inclination with respect to the planet's equator, while outer moons maintain an almost constant orbital inclination with respect to the invariable plane of the solar system. Interesting exceptions are Triton and Earth's moon, both of which have significant inclinations to their planets. Precessions are primarily driven by the planet for the inner moons, and by the sun for the outer moons.

It's probably worth paying some close attention to Uranus and its known moons (and their formation and/or capture) when considering the various thoughts people have put into the thread. Most of the moons and rings have negligible inclination (i.e., they're coplanar with Uranus's equator), and Uranus's extreme axial tilt is intriguing. If the tilt is because of a giant collision, did the relevant moons and rings follow the planetary tilt? Or did they form in the highly-tilted equatorial plane (for some of the moons), or were they captured into the equatorial plane (for others) post-tilt?

Mostly because of conservation of angular momentum: the solar system formed while spinning, and that direction has been maintained. The cases where this is not true (for example, Uranus has its poles "facing the plane" instead of having them orthogonal to it) are likely due to past collisions with foreign bodies that caused the tilt to appear.

Are most of these moons from when the solar system formed?

Maybe not, but the angular momentum is. Without a big input of energy and momentum to cause a plane change, anything that forms in the solar system will orbit roughly in the plane of the ecliptic, and will spin on an axis normal to it.