The risk is small with simple declarative tests ("the output of f(x) == y") but with complicated systems with large state spaces, you start writing combinatorial tests to try to cover more of the space and incur a bigger risk your test code is buggy.
Few things are as frustrating as fixing a bug that should have been caught by a test and realizing the test wasn't actually working correctly. This is even more acute when working on code where speed is important (where performance is correctness, in other words) and you realize a test has been measuring the wrong things, such as asynchronous runtime showing up in the wrong place.
TDD has always seemed more like a useful tactic to be applied to certain kinds of problems than as a silver bullet.
For example, lets say you make a mistake in your test so you are checking (result = expected_result), which is always true, instead of (result == expected_result). Now when you write your code, you run the test and it passes.
In this case your code may or may not be correct, and the test, which contains a bug, does not catch it. But the bug is not a fundamental misunderstanding of the problem, rather a simple mistake in writing the test. Following strict TDD doesn't prevent this.
Actually, following strict TDD does prevent this.
You must see a test fail before you make it pass. It goes "Red -> Green -> Refactor" not just "Green -> Refactor". Even if a valid test case passes on the first time, one should modify the test (or the code) in a known way to cause an expected failure.
Your premise (tests are only as correct as the spec of initial assumptions) is correct, but your supporting example is not.
This seems to be the paradox in much advocacy of strategies like TDD. We start with the premise that our code is likely to contain bugs. In order to detect those bugs, we write a lot of tests, perhaps doubling the size of our code base. Now, we can do whatever we like to the production half of our code base, as long as our tests in the other half all continue to pass when we run them, because magically that testing half of our code base is completely error-free.