I'm not quite sure I am parsing your question correctly, but:

The Moon is yanking the Earth around, but even something that stupendously massive doesn't do much damage (the tidal forces are nothing to sneeze at.. but the 'wobble' caused by the difference between the Earths center of mass and the Earth-Moon barycenter (the point that the Earth and Moon both orbit) is negligible in nearly every consideration).

This would absolutely not be a concern.