Yes, that's why I say "making the test fail is useful" ;)

And yes, I've seen this happen as well, either you're not testing the bug-causing issue or you're using a "Magical Mock Library" where x.does_thing() just returns true and you thought it did the thing but it didn't.