Re:Tarsnap outage (July 2-3, 2023) postmortem
by nodesocket on Thursday July 27, 2023 @11:27AM
(#36895025)
Some recommendations on the AWS front (not sure if some of these are already implemented since the postmortem does not go into AWS details).
- Setup nightly automatic snapshots of EBS volumes (this is supported natively now in AWS under lifecycle manager).
- Use EBS volumes of the new GP3 type, and perhaps use provisioned IOPS.
- Setup a auto-scaling group with automatic failover. Of course increases cost, but should be able to automatically failover to a standby EC2 instance (assuming all the code works automatically which the blog post indicates is not currently the case).
- Setup nightly automatic snapshots of EBS volumes (this is supported natively now in AWS under lifecycle manager).
- Use EBS volumes of the new GP3 type, and perhaps use provisioned IOPS.
- Setup a auto-scaling group with automatic failover. Of course increases cost, but should be able to automatically failover to a standby EC2 instance (assuming all the code works automatically which the blog post indicates is not currently the case).