Even for same architecture compilation, the host's feature set is rarely that of the targets. If you're distributing binaries, you generally want to target a safe, old minimum baseline to maximize user bases--for x86-64, resorting to just sse2 is safe [1] and often sufficient [2]. Most compile farms and developer machines are often newer hardware.

[1] The x86-64 ABI actually requires SSE2 extensions to work correctly.

[2] SSE2 added double-precision and vectorized integer support to the SSE registers, the former allowing you to replace x87 FPU usage for floating point (unless you need long double, which is extremely rare). The newer SSE sets generally add only specialized operations that are unlikely to show up in autovectorization anyways, and the wider instruction set of AVX is less useful for performance in the "it might be useful" autovectorization scenario. They are useful for specific known hot regions of code, and in distributing binaries, variants of these are constructed for different levels of feature sets and are dynamically selected based on the user's actual hardware.