The build time is pretty significant actually. E.g. NixOS takes this to the extreme and always rebuilds packages if their dependencies change. It can take days to rebuild affected packages, and you typically can't easily update in the meantime. This is worse if the rebuild breaks something since someone will have to fix or revert that before it can succeed.

For non-security updates, they can do the rebuild in a branch (staging) so it's non-blocking, but you will feel the pain on a security update.

In practice, users will apply a config mitigation if available or "graft" executables against an updated lib using system.replaceDependencies, which basically search and replaced the built artifacts to use a different file path.