In your fruit basket example, there are two fixes for this. Either improve the public API tests to verify that an hour later the file is still there. Or you use a storage technology where you can assume it will work correctly (Postgres, S3, etc) and then have a code review.
There is definitely an exception to "only need to test the public API". That is when a very complex component has to be built from scratch or almost scratch in order to support the public API. For example, I personally need to build a Lucene-based search server (Solr and ES don't fit my use case). While I shouldn't need unit tests for Lucene itself, I do need unit tests to verify the threading model, file structure, and couple other things I write from scratch are correct because the complexity is so high and I don't trust myself or anyone else to write it correct the first time.
Basically anything more complicated than a "unit" is impossible to test for every permutation and as a result you get bugs when things happen in production that nobody anticipated a test for.
In short, been there, done that. Yes, way easier to write less tests that don't test everything. Witnessed plenty of six figure+ bugs as result.
Meanwhile, if refactoring means breaking a load of tests then you're doing it wrong. TDD is actually pretty hard to learn and it seems to me most people give up when the have to refactor code that they've tested internal details. Given their knowledge they conclude TDD is bogus, rather than wonder if perhaps their tests are bogus.
Genuinely curious - how would you write a lower-level unit test to account for this? If my "testAddFruit" method directly hits the temporally flimsy datastore, how is that any better than the API doing the same, albeit through multiple abstraction tiers?
I am not convinced of this argument. Obviously, it's important to heavily test the endpoints of any API you make public. In fact, that's a situation where you want 100% coverage. However, testing the endpoints of an API does not allow you to forego testing its internals. At best, a passing suite of API endpoints should give you reasonable confidence that the internals are functioning properly.
An API is an abstraction over many moving parts. Any good API will consolidate thousands of lines of business logic into a few endpoints. There is a lot of room for error in the layer of abstraction between API and internal logic. It's entirely possible that a API endpoints could appear to be functioning properly, but actually be relying on broken internal code.
For example, consider a fruit basket API. You can insert fruit into the basket, and check what fruit is in the basket. A suite of tests for the API endpoints could insert fruit, and then check that it's there. In this case, the API is hiding a lot of internal logic. Storage mechanisms, data persistence, fault tolerance, and a slew of other logic decisions are completely opaque to the API consumer.
What if the internal code incorrectly stores the fruit in a temporary file? The API test will pass if it inserts fruit and then checks that it's there. But it is not going to check for that same fruit in an hour. What if it's gone?
Internal logic like data persistence is (rightfully) opaque to API consumers. That means that no testing of API endpoints can validate all internal logic. Therefore, you cannot ignore testing the internal logic in favor of only testing API endpoints.