One of the biggest problems I've seen is how much incorporating secret management complicates deployments. As soon as you add that, you need some sort of encrypted management store (Vault, etc) some workflow to add new secrets, and now you need more refined access control and a lot more work in automation. Simple git push/pull deployments are gone. Default Fabric, Capistrano scripts need to be heavily rewritten and most often this means limiting who can deploy.