You are right, no changes. With a change, most objects are still valid as cached and can result in similar speedup. I'm also wary of caching introducing bugs and intend to continue to CI without it as a safety net. May even require a no cache build before pushing commits.
I'm ok with that I guess and I suppose I see the benefit when running multiple package test runs where their aren't dependencies between them. But I've just not seen the utility in practice, and it seems like it can introduce bugs where the cache is used incorrectly.