If you use a purpose built storage though then you can deny access to the actual keys to just about everyone, so if a developer leaves then they won't have the keys, and if they do you'll know because it will have a built in audit trail. And in many cases you never have to release the key from the purpose built store because it takes in crypto text and gives out plain text without ever revealing the key to the user.

They'll have their own key, which will be revoked from the store, and they might have a single key that machines use to access the store. You should be rotating that machine key all the time anyway, so rotating once more when a dev leaves isn't a big deal. It's also much easier than rotating all of your actual keys, which in many cases is generated by a third party that may make such a thing very hard.