Earthquakes, movement of tectonic plates, etc. can shift the mass of the earth closer to or further from center. This speeds up or slows rotation, like when you are spinning in a chair and pull your arms in. Some energy is probably also lost to tidal friction with the moon.