That's part of the building's history now, for better or worse. I expect Viollet-le-Duc's work will be restored, even if some people think he was crass.
This whole issue isn't something that can be resolved easily. It's important not to look down on the activities of repair and maintenance. They might not be as glamorous as artistic creation, but obviously there's love, skill and creativity involved in repair as well. We need more of that.
A bad intervention by a currently successful artist or architect would be nothing to be proud of in future years. "Move fast and break things" is not appropriate in this situation. It's not as if a repair job involves pure boredom, imitation and humility—those craftsmen will be expressing themselves and their passion just fine.
The 19th C fixation on romantic originality, on the heroic artist? We don't need that. Not to be cynical about the state of architecture, but we can't match even Viollet-le-Duc's iffy work now. We need to shift into a different mode with more emphasis on preserving building skills and historical artefacts than on altering them to suit the whim of the people who happen to be alive now.