doug_neumann

We need read-access to your production environment to survey your infrastructure. Then, depending on the services you're using, we need the ability to backup data, share it with your recovery environment, and cleanup old backups. We make it easy to set this up by giving you a CloudFormation template that includes the roles, policies, etc.

Some of the actions we need to perform can't be completely locked down via IAM policies. An example is the ModifySnapshot API that is used to share EBS snapshots with other accounts. IAM policies don't allow you to constrain which accounts those snapshots are shared with, and we don't want to be sharing your data with any account that isn't yours.

So, instead of asking for ModifySnapshot permissions directly, we include a Lambda function in the CloudFormation template. This function wraps the ModifySnapshot API, and adds validation of the target account. Our role has Invoke permission for this function, but not of ModifySnapshot, thereby eliminating that exfiltration vector.

In the recovery environment, we need more permission to create/modify/destroy the infrastructure that we are managing for you, but we still take a least-privilege approach. We also include a Lambda function in that CloudFormation template to constrain the potentially dangerous APIs.

Thanks, we really do appreciate the feedback. We're gonna run some tests on this.

Well, I'd assert that every system fails eventually. And it's a pretty common practice that you deal with these failures by implementing a redundant system that can take over in case of a failure. Think about drives in a drive array, or redundant web servers behind a load balancer. This is just the same concept applied in the large for an entire cloud workload.

Oops! Fixed that last night and then overwrote the fix with my next deploy. There's a version control lesson in there for me.

It's fixed now.

Our service runs multi-region active across 3 regions. We can survive a 2 region outage and still operate successfully.

Thanks, we really appreciate the support.

We have done a couple of cross-region and cross-account migrations in recent months. It's the same technology but with a different business model. Happy to chat with you about it if you're curious.

Just to be clear, our solution is about failing over all of your environment to another region. So in our case, A, B, & C would all be coming up together from snapshots in that other region. But there are still consistency concerns as you mentioned.

We do everything we can to snapshot servers in quick succession (as much as we can with AWS), but they won't be fully consistent across nodes. We've found, though, that clustered systems like this have built-in capabilities to deal with these inconsistencies. It's kinda similar to if you lost power to all 3 systems and then brought them back up - they might not all be at the exact same point in time, and the application would need to sort that out.

If you'll tell me what application this is (is it a database platform?) I can do some quick research for you.

Okay, thanks for the feedback, we'll definitely spend some time talking about it.

Spot-on with the multi-cloud story. But we don't plan to wait a few years :-).

I like to think our vision is bigger than that. Amazon offers a lot of building blocks. We're assembling those blocks into a solution so that AWS customers don't have to assemble the blocks themselves. As Amazon changes and enhances those blocks, we'll evolve what it looks like to build on top of them appropriately.

And before long, we envision offering analogous functionality for the other public clouds. We've found that the most valuable customers are using multiple clouds and want a single solution across them. So even if Amazon eventually solves this problem for AWS, the multi-cloud market should offer us plenty of opportunity.

Gotcha. I sure hope that NDA'd response is "we're about to unleash 'Cognito2' that rights all the wrongs of the current Cognito."

Have they ever given you a response? Cognito has so much unrealized potential...

Not yet, but they're on the near-term roadmap.

We've helped customers manually setup bucket replication for S3 when they've needed it. The plan is to eventually automate that, including backfilling existing objects as needed.

WRT lambda, we roughly want to conquer virtual server workloads, then containerized workloads, and then we'll hit serverless workloads. Caveat being that everybody's "customer driven" these days and we can easily shift priorities if people really need lambda support.