I believe the shift in meaning came about because decimation was an incredibly brutal punishment -- it was the most severe sanction that a general could impose.

What is often overlooked is not just that it was "one in ten" executed, but that they were executed by being clubbed to death by the other nine men who were not chosen to die.

So "decimation" spread from the literal procedure to "worst outcome imaginable". Much as "literal" no longer means literal.