You really want to avoid a dependency on prior content of the destination register, to allow renaming and maximize out of order scheduling.

It's to break dependencies for register renaming. If you have an instruction like

  mov w5, w6 // move low 32 bits of register 6 into low 32 bits of register 5

This instruction only depends on the value of register 6. If instead it of zeroing the upper half it left it unchanged, then it would depend on w6 and also the previous value of register 5. That would constrain the renamer and consequently out-of-order execution.

I don't know either, but why wouldn't backwards compatibility apply to aarch64? It too is based on a pre-existing 32-bit architecture.

I don't think it's backwards compatible the same way x86-64 is.