If this leads to having to spend less time on writing tests in total, things have gone right.

These fakes are much less maintenance heavy. As changes to implementation details don't require changes to tests. Hence, whilst you need to write tests for your tests, you spend less time writing tests in total.