From that point on, the build may become "fully unrepeatable" because previous steps may have reused cached stuff that can no longer be reproduced.

You could break the build simply by cleaning some things from the cache, without changing the Dockerfile, even thought that Dockerfile "succeeded to build"