Assuming the cave is spherical, each swing will reach the other side, because inside the cave the black hole will not be slowed down, do you agree? So in this perfect situation, it is impossible that the black hole will come to stop exactly in the center. Even if it was stopped exactly in the center, if one atom hits it from one side, the black hole will crash into the other side (but slowly of course).
I see your point (I didn't before).
There is no force on the black hole, so if it moved at all it would crash into the wall. Only when it actually ate some of the wall would it turn around. So each swing would eat more and more.
Imagine a tiny black hole arriving on earth. It grabs some mass. The black hole is moving, but the mass it ate is not. So rather than free falling, it looses momentum. As it falls through the earth it keeps doing that.
So it will not rebound back up to the top at the same height it fell. It will be lower. The second time through the same thing will happen.
Each time it oscillates it will get lower and lower.
Next. Say the black hole is a bit bigger. And it eats some mass on one side of the earth - now there is an imbalance. There is more mass on the other side. So the forces will cause it to slow down more than it otherwise would.
When it finally settles down enough to make a cave, it will "swing" as much as it can. But each swing can not be any higher than the one before. Over time, with random atoms falling into the cave it, it will come to rest at the exact center.