Indeed! I recall a lexer and parser built via TDD with a test suite that specified every detail of a DSL. A few years later, both were rewritten completely from scratch while all the tests stayed the same. When we got to passing all tests, it was working exactly as before, only much more efficiently.

From that experience, I would say that in some contexts, tests shouldn't be removed unless what it's testing is no longer being used.