The point of GPL is that you own the entirety of the source with your changes. Remove your changes and you're back to the original source which everyone "owns."

"Owns" in quotes because that's not ownership, it's a license to use, modify and distribute the code within certain limitations. I've been dealing with IP lawyers recently over exactly this issue and they're being sticklers over the difference.