Launch HN: Arpio (YC W21) – Protect your business from AWS outages, ransomware
What that means is that when critical AWS services go down (like the Kinesis outage in November [0]), Arpio can launch identical workloads in a healthy region. Or if a bad actor does bad things in an AWS environment (like Codespaces [1] or Webex Teams [2]), Arpio can quickly restore everything to an alternate AWS account.
Our story goes back to the big S3 outage of 2017. In February that year an AWS employee made a typo at the command line, and inadvertently took down much of AWS’s Northern Virginia region. That outage lasted 5 hours, and we were among the thousands of companies impacted.
All outages suck, but the timing on this one was particularly bad for our business. And worse, we had no control -- all we could do was wait for Amazon to get us back online. As you can imagine, the execs weren’t exactly happy about that...
With Arpio, we’re building the solution we wish we’d had back then. Arpio maintains an exact replica of your production AWS environment in a different region (that you choose) and optionally in a locked-down AWS account (that you own).
This recovery environment includes your data and your infrastructure, and it’s updated frequently as your environment evolves. It’s also checkpointed, so you can roll back to a prior state to recover from data corruption or ransomware. And when you aren’t using it, it’s dormant, so you don’t have to pay AWS for resources you don’t need.
But when you need it (or want to test it), Arpio can have it up and running in a few minutes.
Disaster recovery is usually custom-engineered for a given workload. With Arpio, we’re building a general-purpose solution that works for most AWS workloads. We handle the complexity ensuring every route table is rewired, every security group rule is correct, every private IP address is preserved, and every database hostname is aliased. And handling that complexity makes Arpio simple to implement. We can often get new customers onboarded in under an hour.
Arpio works today with EC2, EBS, RDS, ECS, ECR, ELB, VPC, IAM, ACM, Autoscaling, Cognito, ElastiCache, and CloudWatch. We’re delivering Beanstalk and EFS support in the coming weeks. If we don’t yet support your environment, drop a comment below - we’d love to get your feedback on what we should build next.
We encourage you to take it for a spin. Or if you’re up for a chat, send me a note (doug[at]arpio.io) - I’d love to walk you through it in person.
So, HN, what do you think? We’re excited to get your feedback!
Thanks, Shaw & Doug
[0] https://aws.amazon.com/message/11201/ [1] https://news.ycombinator.com/item?id=7909791 [2] https://news.ycombinator.com/item?id=24319293
That’s good. I’m wondering how you support cognito though?
Do you just run custom ETLs in customer lambdas?
And wouldn’t this cause all users to have to reset their passwords in the new pool?
Looks really interesting and definitely does things different than “run terraform with a different region”.
The experience is better for user pools that integrate with external identity providers, like SAML, since the IdP metadata can be fully replicated into the other region ahead of time.
AWS makes a simple move between user pools inexplicably difficult. They don't even offer an export and import in the same file format as far as I know (bulk export is all JSON via the CLI and import is CSV, but maybe I missed something).
This is ridiculous in my opinion given that once a pool is configured you cannot change attributes (say, turning on a middle name field).
We were just discussing Cognito being a pretty significant single point of failure in our current setup (any vendor would be, though). I wish they would offer replication as the other commenter mentioned.
We've helped customers manually setup bucket replication for S3 when they've needed it. The plan is to eventually automate that, including backfilling existing objects as needed.
WRT lambda, we roughly want to conquer virtual server workloads, then containerized workloads, and then we'll hit serverless workloads. Caveat being that everybody's "customer driven" these days and we can easily shift priorities if people really need lambda support.
It's fixed now.
We do everything we can to snapshot servers in quick succession (as much as we can with AWS), but they won't be fully consistent across nodes. We've found, though, that clustered systems like this have built-in capabilities to deal with these inconsistencies. It's kinda similar to if you lost power to all 3 systems and then brought them back up - they might not all be at the exact same point in time, and the application would need to sort that out.
If you'll tell me what application this is (is it a database platform?) I can do some quick research for you.
Some of the actions we need to perform can't be completely locked down via IAM policies. An example is the ModifySnapshot API that is used to share EBS snapshots with other accounts. IAM policies don't allow you to constrain which accounts those snapshots are shared with, and we don't want to be sharing your data with any account that isn't yours.
So, instead of asking for ModifySnapshot permissions directly, we include a Lambda function in the CloudFormation template. This function wraps the ModifySnapshot API, and adds validation of the target account. Our role has Invoke permission for this function, but not of ModifySnapshot, thereby eliminating that exfiltration vector.
In the recovery environment, we need more permission to create/modify/destroy the infrastructure that we are managing for you, but we still take a least-privilege approach. We also include a Lambda function in that CloudFormation template to constrain the potentially dangerous APIs.
And before long, we envision offering analogous functionality for the other public clouds. We've found that the most valuable customers are using multiple clouds and want a single solution across them. So even if Amazon eventually solves this problem for AWS, the multi-cloud market should offer us plenty of opportunity.
Joe Arpio is a convicted felon[0] for contempt of court and treating humans in a sub-human way[1].
I strongly urge you to reconsider this.
[0] https://www.phoenixnewtimes.com/news/sheriff-joe-arpaio-gets...
[1] https://archive.thinkprogress.org/twenty-years-of-in-tents-t...
And apologies for not saying this before: heartiest congratulations for your launch, I wish you nothing but a resounding success, whatever the name may be.
As our friends on WSB say - rocket emoji and moon emoji!