A glass bead at the bottom of a bowl is in a stable equilibrium, if you shake the bowl side to side it'll come back at the bottom. A basketball spinning on your index finger is in an unstable equilibrium, it stays there while the conditions are met and any small deviation will make it fall
L2 is called unstable above. So orbiting around it instead of hovering immediately at it should be like trying to spin a ball slightly off the tip of your finger. E.g. you wouldn't want to do it.
Sibling replies have proper explanations.
https://en.wikipedia.org/wiki/File:Lagrange_points2.svg
The idea of plainly stable and unstable points only happen if you have rotational symmetry, and doesn't really apply to most of the points. Leading the discussion on those terms isn't productive.
It orbites in a plane where it's always subject to attraction, but that doesn't mean the point always attracts the satellite.
Like I said, since you can't place a satellite perfectly in practice, its motion will always be a combination of the 3 modes, and eventually the unstable mode will dominate and the satellite will move away from L1/2/3. BUT, if you get the math right you can position the satellite so that most of its motion is in the oscillatory mode (i.e. orbiting the point), and then as it drifts into the unstable mode you nudge it back to the oscillatory mode with a thruster firing.
For context, the amount of thruster firings necessary to keep the ARTEMIS probes around Earth-Moon L1/2 was about 5m/s per year.