I was wondering how it would compare to the code produced by auto-vectorization, I guess it doesn't matter with hand-crafted assembly.

The compiler can't autovectorize because it can't assume it's okay to read past the end of the string. Only the human or the machine can make that determination, using special knowledge about the environment, such as that when locating the NUL byte out-of-bounds reads are okay so long as they don't cross a page boundary. And "okay" is a stretch because tools like Valgrind ship with huge lists of manually maintained suppressions to account for hacks like this that violate the standard rules.

But in this case a human applied their special knowledge in glibc, what's stopping the human applying the same knowledge to the compiler itself for this specific OS/arch?

Primarily because it would make it more difficult to detect and debug out-of-bounds reads, so at the very least it's not something you'd want to do by default, even at high optimization levels.

Possibly it might also be an awkward, complex, or dangerous (as in risk of unintended consequences) optimization to selectively violate the memory model that way. But I'm not familiar with the internals of optimizing compilers, so that's just conjecture.