I like this a lot! My root README ends up looking a lot like your SPEC.md, and I also have a file that’s pretty similar to your TODO.md.

My experience agrees that separating the README and the TODO is super helpful for managing context.