Cache files are frequently non-trivial in size and go into ~/Library/Caches, and those are not cleaned up automatically.