I’ve been bitten by it doing something as simple as creating parameters using CloudFormation. The only way around it is judicious use of DependsOn to make them single threaded.

Thanks for this, I was considering bringing params into CF, the main blocker being that it doesn't support secure params (which is obvious in retrospect, they wouldn't be secure in the template).

For secure strings in cloud formation, I use a cloud formation parameter ( - meaning I have to enter the parameter manually and set NoEcho to true. And then set the CF parameter value to the SSM Parameter value.

This is with the help of a lambda backed custom resource

https://svdgraaf.nl/2018/04/13/CloudFormation-ssm-secure-str...

Obviously , this can’t be a part of your CI/CD Pipeline. I run the CloudFormation template manually in the console to enter the parameters and my CI Pipeline can then update the stack when needed.

Fetching parameters by path is an easy way to get bulk parameters and avoiding the rate limits.

It has caused us failures as lambdas have scaled up as it is quite easy to hit a rate limit.

For me, I was able to reduce the calls from lambda pretty significantly by doing the parameter pull step outside of the handler, so it is available as part of the execution context between runs (similar to how many people do DB connection with lambda). I still do the decrypt step within the handler though because I don't like the idea of having decrypted values lying around, but the limit for KMS decrypt operations is independent of SSM, and is a 10k/s shared limit in the regions I use, so I haven't had any issues yet.