And when I code in C, there's typically at least one library with inline assembly even though my C doesn't use any inline assembly.

There is indeed a difference here between the vast majority taking part in risky behavior and library authors using focused performance improvements. There are a lot more eyes on the library code due to downstream effects than in everyday top-level code.