It can be, I won't entirely disagree. But it depends how highly coupled your unit tests are to the design.

Unit tests are a safety mechanism- you can easily bog yourself down with too many, tying things up in bad ways, or presuming a specific design. But do you really want to live with no safety at all?

The art, as I see it, lies in writing tests in such a way that you get the guarantees you need, the safety of not letting the next guy (or yourself) screw it all up during later changes while simultaneously not preventing you from doing needed refactoring.

It's a fun challenge.