How long do you keep the transaction logs before rewriting them?

I too had a few EC2 instances go down with signs of being severed from the EBS in the recent couple of weeks; mine were in eu-west.

There's a continual background cleaning process which depends on the amount of storage which can be reclaimed -- there's a tradeoff between cleaning too slowly (and paying for wasted storage) and cleaning too fast (and paying for lots of S3 operations). I think it averages a couple weeks right now.

Why would I use your service over restic?

God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

You really shouldn’t if that’s a major concern for you and that is a valid concern. For the same reason I’ll never use PurelyMail otherwise it’s perfect.

I know you didn’t ask me — but I don’t think Colin can answer differently other than saying that he is training a family member or friend to take over if needed.

Here’s more https://news.ycombinator.com/item?id=7514753 this is also linked there http://mail.tarsnap.com/tarsnap-users/msg00846.html

Very old threads but I am not sure much has changed there https://www.tarsnap.com/contact.html

Why would you use it instead of restic? Well, for pricing in pico dollars ;-)

and for it has a functional GUI with tiny system footprint and that there really aren’t many such solutions out there.

I'm curious how the prices shake out against services like Wasabi, since it's just dumping to an AWS S3 bucket

Wasabi does $7/TB with no ingress/egress fees. My NAS is set up to rclone to it about once a day and I've yet to have any problems

I haven’t checked the pricing in a long time but you can use Tarsnap also if you have to backup only 7.3kb (okay I might ne exaggerating here but you get the drift) and pay for only that much. You can’t do that with Wasabi et al.

Also it’s really simple and does what it says it does, nothing more, nothing less. In today’s everything convoluted and bloated world this is a luxury imho. The GUI app is also quite good and functional. Support is prompt (that is if you need it).

You don’t have to worry about file being deleted just because your machine didn’t connect or backup for some time even if you keep paying (hello Backblaze) etc. I mean there’s no circus, melodrama , and cliffhangers involved.

I personally would never use it backup my entire laptop, due to price alone. But I have a subset of VVI files and Tarsnap is one of more than one backups for those files. So for that use-case Tarsnap is perfect for me, so far.

Backblaze has kept my ‘shutdown two years ago’ machine data without issue. What problems did you have with them (or did others have)?

Backblaze has a policy of allowing backups of external disks, but the disks have to be connected at least once every 30 days, or they'll delete the backups. I understand they want to avoid abuse, but the lack of any grace period, or ability of support to ad an override, really soured the service for me.

Huh, I had to start paying them $2 more for my nonexistent PC I think, but otherwise was fine. I have only 1 TB of total storage on that PC though, so maybe that’s the reason.

You can just pay extra for extended or infinite retention. https://www.backblaze.com/cloud-backup/features/extended-ver...

I did this, it's come in use a few times..

What use is SLA? If a service goes down for too long, are you really going to hire a lawyer sue it over SLA or just... use another backup?

Not even then - most SLAs say that if it's breached, you pay less. Not that you get money back

It's not about suing, but defining expectations about how you can rely on a service.

For example, my team has people across the world for HW bringup, so we can't allow our code hosting or CI to be down for more than a few hours. Of course, backups have different uptime requirements, but as for everything, it's a tradeoff between features, of which an SLA is one.

Tarsnap's features are granularity of cost, reliability of storage, and encryption, but not 99.999% uptime.

> It's not about suing, but defining expectations about how you can rely on a service.

Meeeeh, my ISP cut of around 100+ fiber connections in my town and spend three weeks fixing it. My neighbor have business line, there's an SLA on those that among other things, require them if reestablish his connection within 3 - 5 hours. It took them over 500 hours, so that SLA is useless for anything but forcing compensations.

The problem is that the SLA should give an indication of available resources, but in reality it's mostly a contractual thing for most companies, they'll pay the "fine" or refund a customer if they fail to hit their SLA and that's about it. Tarsnap most likely have better availability than many midsize competitors simply because it's just one person who really cares about it. Doesn't help if he's hit by a bus though.

The "problem" is that no sane company will sign for any damage compensation on some cheapo few dollars a month service.

A company could... if you have N users and you pay M for storage per user and downtime cost you X then it could be that a discount of Y means (M - Y) * N = X

SLAs can be meaningless like that. However the better ISPs have in place a backup system that doesn't use the same fiber/wires. Sure the backup might be a radio or satellite feed and so be slower, but it will get/keep you online. This costs are lot more per month though, so if you are not paying for that service your SLA will probably just be we give you a free month (which hurts them enough that they will do some things to prevent downtime, but not enough that they put redundant fiber paths in the ground)

Agree. You get a discount if something breaks. But SLA really only works for larger services where the cost of fixing something is small when compared to the discounts.

Then SLO (service level objectives) should be enough.

huh, never heard of SLO before.

and now also go google what SLI means ;-)

Very roughly:

SLI - Service Level Indicators - Metrics ie Latency of each request / response cycle

SLO - Service Level Objective - What threshold we are aiming for - 10 ms from request to response averaged over 1 hour period.

SLA - SL agreement - contract with custom yet what happens if we breach (credits given, put the CTO in stocks and throw eggs at him etc)

I know it's a joke, but I think if an SLA involved putting a CTO in stocks and throwing eggs at him then that'd encourage me to sign up for the service. Especially if the video of it were posted after every incident.

Instead we get refunded some pitiful amount when our business is seriously disrupted for an extended period of time.

:-)

My youngest once found some sort of chocolate drops called "unicorn poo" - which seems a more ironic thing to chuck at CTOs !

Don't let the CTO be a scapegoat. Entire executive leadership, all board members and the 5 largest shareholders.

We have just written the Sarbanes-Oxley for the tech regulation industry- all we need now is a congresswoman and a senator and a good acronym

Secure

Technology

Oversight for

Corporate

Software

STOCS Act here we come !

Edit : yeah I could not get the K in ... that's hard

Korporate.

If the data is super important you should be setting on two different providers anyways for backups.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it.

What is this mythical unimportant data that people still want to back up?

My nginx config is not super important. My old reports written for study are not super important. My pirated movie copies are not super important.

These are examples of data that I could easily live without. Where losing it would either be a matter of re-doing old work, or just forgetting about old and minor things.

The stuff I don't want to fuck around searching re-downloading from torrent for example.

Anything that you stashed just for convenience, but you could re-download or re-generate it if really needed, or simply live without it... frankly, like 90% of stuff on my disks fall in the category "I'll read/view it one day", which in reality I'll probably never have time or patience to open ever again.

Strange, 90% of the things trapped in my flash memory are system files.

You should get another drive and reserve it to Data. The cost is negligible and it really makes everything much simpler.

Optimizing your system or upgrading it just becomes a "trash boot drive and reinstall" operation, applied without a care in the world.

I really used to enjoy formatting my machines about every 6 months.

Well I used to until macOS kinda went off the rails a bit. Now it’s mostly an exercise in running my arch script for my thinkpad.

Being stuck between operating systems is kinda a mess though, makes backup and file sync in general really hard. But everyone’s gotta have their own cloud, right?!

Why can’t I just put a cloud under my bed and forget about it?

> Why can’t I just put a cloud under my bed and forget about it?

Just buy a Synology NAS. Keep default settings, set up a few user accounts, tweak a few things here and there, enable encryption, install Active Backup on all your devices, done.

There are many cheaper/more open options for self-owned NAS storage, but contrary to a Synology they're definitely far and away from "and forget about it".

I mean, you called it crap and then said it's super important. That's what hoarders say.

Subjectively you may feel that your data is super important, but objectively it probably isn't.

When people talk about 'super important' (totally a technical term), I think of things like DB backups in software companies, backups of financial reporting for firms, etc. Not your tax return from 2008.

>What is this mythical unimportant data that people still want to back up?

I have lots of stuff like this. Often it is easier to just back up an entire folder than go through sub/sub folders separating stuff into: important, not very important. Storage costs are low enough to just backup everything (almost). Also, one often doesn't know what may be important/useful in future. For example a couple of years ago I had this huge buildroot system (600gb) to build firmware images for a single board computer I spent quite a while to put together. The project I was doing it for got cancelled so I had no need to keep it. Still I wish I did, as I'd love to be able to tinker with it now, but 600gb is not a trivial amount to store so it got deleted. Most of this data was pulled from various online resources that don't exist anymore too.

What's the morale of my story? If you have a fast internet connection (I don't) backup "everything" to cloud. Then find "really important stuff" like the pictures of your children etc and back it up again to a different cloud.

If you're in a middle of nowhere on a slow LTE connection like me, building a nas box is not a bad idea for backups.

I have stuff like that on a hard drive in my home, on a persistent storage volume from Linode, and on Dropbox.

tarsnap natively protects against inadvertent or malicious deletion or corruption — old tarsnap backups are immutablez The low-cost competitors (restic, borg, etc) seem to have this feature as an afterthought, and they make it surprisingly difficult.

(FWIW, S3 can be somewhat straightforwardly configured so that old data is effectively immutable. Google Cloud Storage’s similarly named versioning feature appears to be far weaker.)

Yep, S3 is reasonably easy to configure for immutability. I personally use restic to send (encrypted) blobs to https://www.borgbase.com which has append-only mode and monitoring to warn me if some backups didn't happen.

borgbase is another “little” service that I use and like just like tarsnap and to some extent rsync.net. And they also have an excellent gui app Vorta (it’s FOSS; borgbase dev is the maintainer).

It's a pretty well-known fact for years that tarsnap is basically a one-man show, and yet Colin has managed to provide fantastic service so far. Sometimes having ppl who built the service also managing it is actually a big plus, compared to other services where you first have to fight through outsourced & underpaid support that's limited to template answers, only to finally get some "engineer" who got that job 2 months ago and is more clueless on their system than myself...

Why the scare quotes? I would expect any well-experienced power user to know a complicated system better than a fresh engineer two months into working on it, with no previous experience on the system. Especially if the power user is an engineer themself.

I hate to bring this up, but what about the bus factor? If Colin is physically unable to continue maintaining the service and something like this happens again, how will anyone be able to get their data out? It's not really a concern about the service Tarsnap provides today

There's an old Sys Admin saying (perhaps from Allan Jude of ScaleEngine) that goes something like "if your data doesn't exist in at least three places, it doesn't actually exist at all..."

That is to say, if Tarsnap is the only place you've keeping sensitive/important data, then you're "not doing it right" as a backup. Things happen... your hard drive can die suddenly, and a data center bursts into flames all on the same day.

I feel like ovh will never stop earing about this. This has been, frankly, a traumatic event for many sysadmins I believe, and one that was shared by many from the same source, which is quite different from the standard variation of "that time when I erased the production database" (looking at you gitlab, but also at myself!). I mean, at this point it's between a legend and a warning tale and I don't know what else to call it. A bad Wednesday probably.

> I feel like ovh will never stop earing about this.

To be fair, they deserve it a bit as they got up in flames twice .

Indeed, after the first fire, the geniuses over there collected all the UPS and batteries they could find from the DC and stored them all in a pile in a closed container... where they predictably bulged, failed, sparked and eventually triggered another fire after a couple days.

And to be frank, I've seen plenty of mission-critical services at $bigco which may have had a team of engineers working on them, but the core functionality was maintained, understood, and supported by effectively one senior engineer. If anything went wrong, the supporting junior staff might have been able to fix reasonably simple stuff, but there was effectively one person who understood the system deeply enough to handle problems of any real significance.

* * *

Even large organizations can have fairly regular availability issues. I appreciate the noted flaws of "single point of failure", but I also see orgs where 100s of people have access to the infrastructure, make a change, and then it breaks something. I wouldn't do business with an org just because they have many people, that won't mean they're operationally sound, at least not to my expectations.

> God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service

Hence the toddler.

Are you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!

Raising the toddler to have some help running the business.

Not just help: there is now a clear heir to take over if (the gods forbid) cperciva ever succumbs to illness or is defeated in battle.

Might take a while. Tarsnap has never had an employee without a doctorate. She's a very bright girl but I'll be surprised if she gets her doctorate before 2040.

So you're the one in charge of the unix epoch rollover?

I am really confused by the communication thread and am interpreting that the toddler is somehow in charge of running the infrastructure as a joke. Yet I can’t see it as either a joke or serious.

I’m a native English speaker but sometimes I swear I’m losing grasp on communication in the Internet age and am sincerely trying to understand this all.

The joke is that the toddler is for future maintenance, not now.

Uptime isn't an important property of a backup solution, so I'm not sure where the expectation comes from?

It sure should be up when you need it, exactly at the time you need it.

This was an extremely well written and thoughtful postmortem, but I hope to never see one from you again. :)

It was a postmortem without the mandatory "how can we prevent this in the future" steps…

I think that's a little unfair given what was in the postmortem. It may not be a separate section with the key points, but the information is all there of what the issues were and what the solutions are. I think it's fair to assume they're actually acting on those without them needing to be reiterated at the bottom of the page.

I agree, we don't really need a "key points/future actions" section that boils down to "The service will be geo redundant"

Well, for sure he has fixed several bugs, but he didn't say that he would be testing his disaster recovery procedure every year in the future for example.

Yes, rehearsing the process every year is the main lesson learned. Sorry, it was getting late and I wanted to get the email out so I cut it short.

In 15+ years of running this service, this is one of two (2) postmortems he's ever published, and the first in eleven (yes, 11) years.

Are you gonna switch to us-east-2?

Thank you for the post-mortem Colin and I hope you get some sleep!

Thanks, I did! My long suffering wife was up at 3:30 though. :-(

Time to get your toddler providing round-the-clock support! ;)

Have been having some luck reading https://www.amazon.com/No-Cry-Sleep-Solution-Toddlers-Presch... - available everywhere libraries (blockbuster for books!) are found.

She's generally a wonderful girl. Right now she's dealing with her second molars coming out and just picked up a cold though, which is throwing off her sleep schedule.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

It always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve.

A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.

That's an overly cynical take, post-mortems are not for anyone's reassurance, they are a learning opportunity.

The airline industry is as safe as it is because every accident gets thoroughly investigated with detailed reports ("post-mortems") including what to do differently going forward. These are taken as gospel among all players in the industry and as a result, you very rarely see two different accidents caused by the same thing anymore.

That was entirely not what I was getting at and is a cheap shot that is well beneath you, especially because I suspect that you know that that wasn't what I was getting at.

My comment wasn't intended personally; your words about "will never recur" just reminded me of this peculiarity of software systems, where it's often error handling/monitoring/backups/etc. that cause cascading failures in the systems they're intended to safeguard.

I'm sorry if I misconstrued your meaning, but I am flattered that you think there are things beneath me!

Fair enough. I see the whole function of a postmortem in a very simple way: to avoid recurrence of the same fault. Yes, there will be plenty of new ones to make. But if you don't change your processes as the result of a failure you are almost certainly going to see a repeat because the bulk of the conditions are still the same. All it takes then is a minor regression and you're back to where you were before. This I've seen many times in practice and I suspect that Colin isn't immune to it. And yes, I look up to you, your writing is usually sharp and on point and has both amused me and educated me. So you have an image to live up to ;)

You'd love my team's recent postmortem, featuring the comment "action items haeve been copied from the previous postmortem".

Could be as simple as "test restore a new server every 1-2 years"

Yeah, I was going to do that but it was getting late, I wanted to get some sleep, and the post-mortem had already been waiting far too long to be sent out.

The main lesson learned was "rehearse this process at least once a year".

Agreed, that's the big one. But also: when sleep deprived: take a nap!

You should consider this possible lesson:

"Our simple model that fails gracefully did so and was simple to recover"

Redundancies and failsafes are not free - they add complexity.

99.9% availability fails in boring ways.

99.999% availability fails in fascinating ways.

The infrastructure page* says,

> at the present time it is possible — but quite unlikely — that a hardware failure would result in the Tarsnap service becoming unavailable until a new EC2 instance can be launched and the Tarsnap server code can be restarted ... So far such an outage has never occurred

I read the postmortem as that a hardware failure did cause it to be unavailable and the code could not be restarted, a new server had to be built. If my assessment is correct, lessons (as Jacques mentions) could be written up and this page updated with outage information. I think one outage of a single day in fifteen years is impressive. If my ballpark math is correct, that's 99.992% uptime, ie four nines.

* http://www.tarsnap.com/infrastructure.html

Some recommendations on the AWS front (not sure if some of these are already implemented since the postmortem does not go into AWS details).

- Setup nightly automatic snapshots of EBS volumes (this is supported natively now in AWS under lifecycle manager).

- Use EBS volumes of the new GP3 type, and perhaps use provisioned IOPS.

- Setup a auto-scaling group with automatic failover. Of course increases cost, but should be able to automatically failover to a standby EC2 instance (assuming all the code works automatically which the blog post indicates is not currently the case).

Can you say a bit more about the log-structured S3 filesystem? I wrote something very similar recently (https://github.com/isaackhor/objectfs) and I'm curious what made you settle on that architecture. The closest thing I know of that's similar is Nvidia's ProxyFS (https://github.com/NVIDIA/proxyfs)

> the central Tarsnap server (hosted in Amazon's EC2 us-east-1 region)

What prevents you to distribute load among other regions?

(Also: did you ever think about abandoning AWS?)

Nice write up. A couple questions:

- The use of “I” begs the question: what’s the “bus factor” of Tarsnap? If you were unavailable, temporarily or permanently, what are the contingency plans?

- Will you be making any other changes to improve the recovery time, or did the system mostly function as designed? For example having a hot spare central server?

What I'm wondering is, I had data on Tarsnap, why am I only hearing about this now?