Funnily enough, they're two different Latin word elements. Simplified, there's the "down" de-, as in "destroy," and the "undo" de-, like "defrost." The "undo" de- is also the "not" de- and the "away" de-, related to dis- ("disallow", "discard").