It's a good question! It's in fact such a good question that the reproducible builds folks had to draft a solution for it: the SOURCE_DATE_EPOCH specification. Reproducible builds read the timestamp of the last debian/changelog entry (you can imagine the last git commit or something, for non-Debian packages) and pass it through to the build system, so that the "current" time in tarballs etc. is actually the time of the changelog entry.

https://reproducible-builds.org/specs/source-date-epoch/

For things like build numbers, there's a tool called strip-nondeterminism that cleans them up:

https://packages.debian.org/sid/strip-nondeterminism