Hard agree. I think it's easy to get stuck on the mental model that "the configuration value is the secret" (which means the configuration itself needs special handling), and the correct mental model should be "the configuration value is the identifier of the secret", with another system (Azure Key Vault, etc) being responsible for converting the secret-identifier into the secret-value.

My company open sourced the .NET library we use for this: https://github.com/Lokad/Secrets