It turns out that the magic thing we have to add, exactly fits the observations we have of electromagnetism. That's an indication that we screw with the theory in the same way that nature does, and in the end that's the goal of physics: understand the way nature behaves.