I think you don’t even need to redeploy any long-running tasks. If they have a connection open, it should continue working, and if they don’t, they should consult the vault when they open one and get the new credentials from it (this may be different for different kinds of credentials)
There will be a race condition between creating credentials and storing them in the vault, but even that can be avoided:
- create new credentials with same rights as the old ones
- update vault
- delete old credentials
Example, if it's AWS it may rely on ec2 instance role, to allow it access to the secret manager.
If it's kubernetes, it can be done through k8s token mount, basically allow token in namespace access the vault, and the token(which is generated and manage by k8s, which is just a JWT btw) is mounted into your pod.
I'm not the person to ask how to set it up but it's seamless in use and can return data in json format which makes it easy to parse. If it were my money I'd look very hard at this product.
(I should also point out that this is in a batch ETL system.)
The nice thing about this approach is that it is virtually impossible for secrets to end up in source control. The risk is that the vault becomes a major single point of failure.