I agree, I think it has to do with the motion filtering of your retina. If the rotational speed is faster than the linear movement, the motion detection circuits on your retina will detect movement predominantly in the rotation direction. On the other hand, when you fixate on the ball, the brain instead judges motion by eyeball movement, and the illusion disappears.

Very cool.