I had a terrible, terrifying experience with DO where they hard locked my account for days (and it would have been weeks if I hadn't made a huge stink) because of a misunderstanding. They ban/lock first, ask questions later, and that's not cool and I can't have my (and my customer's) important infrastructure treated that way.

I now use Linode for most of my stuff, and once they have cloud firewalls available in Dallas I'll be moving the rest of infrastructure over as I can.

I really need to write up my story into a blog post I can link to, but until then, I put a recap here a few days ago for anyone interested in the details. Mine wasn't as bad as some people's, but pretty scary personally: https://news.ycombinator.com/item?id=25806086

I’ve had only positive experiences with Linode, including incidents where I faced absolutely massive DDoS attacks.

On the other hand, my account is suspended from DigitalOcean, I have multiple droplets they refuse to give the data to, and they even had the audacity to keep billing me until I did credit card chargebacks. I still have no idea why I was banned.

Maybe because I used the GitHub student pack promo but I had already finished uni? I don’t know.

Another happy Linode user here. They might not have the nicest interface, but their service is top notch and I much prefer their UI to Digital Ocean's any day of the week. I've been with Linode for ten years now.

> They ban/lock first, ask questions later

Given the sheer volume of dodgy packets that originate from DO network space, I find this surprising.

I guess I’m not terribly surprised. It fits a narrative, not of being overly protective, but of run-of-the-mill incompetence.

Someone once used a DigitalOcean droplet to run an aggressive layer 7 denial of service attack against a service I host. I notified DO via their web form, didn’t get a reply for three days, then got told to email to their abuse@ email address instead.

Dutifully, I mailed abuse@ and included the relevant server logs, gzipped, as an attachment. I got a reply that they won’t open any attachments, please copy and paste the logs directly into the email.

As an outsider, it seems to me as if they have few people with the appropriate technical background staffing their abuse department, so legitimate reports are left uninvestigated due to dumb technicalities like not being willing/able to open attachments, and bogus reports are accepted without thought or understanding as to the appropriate course of action.

They're definitely on the theater side of security. I've got locked out of my account multiple times for no reason.

Furious the second time, I sent a crafted image of an invalid ID to check if they actually even care. It seems they don't, as the account got unlocked all the same.

I think you should make a blog post with screenshots calling them out for this ridiculous hypocrisy.

Of its own the blog post won't do anything, but the next time they're being overeager and shutting down a legitimate customer, you can link to that and corner them to fix their procedures.

This is the most offensive part to me, I can't even sign up or use DO without having my privacy invaded and sold (they load multiple fingerprinters from different third party vendors that also then share this data) yet they still don't have a handle on abuse.

I get so much rubbish from DO space, nonstop port scanning at absurd volumes (sometimes totaling 6 digit+ pps), from customers that haven't been removed in years. massscan/zmap from people with a fake opt out page (that I shouldn't need to be opting out of), etc.

They flagged my account and pulled 100% of my servers with no mitigation for 12 hours until I convinced them I was legitimate. DO still handles that incredibly poorly, which is the reason we no longer host anything critical there.

Well, this is a discouraging news. Where have you moved your critical operation?

For clarity, is "They" DO or tarsnap?

tarsnap doesn't rent servers

Tarsnap uses AWS S3

Also both tarsnap and S3 have no idea what content they’re storing on your behalf. (Everything is encrypted at rest). So for tarsnap, deplatforming should never be an issue.

I’ve been hearing about this happen for a long time. Unfortunate that it’s still the case.

What or who do you suggest instead ?AWS?

Vultr. They’ve been fantastic.

DO will also use "machine learning" and "AI" to shut down your machines or terminate your account if your usage pattern is disliked.

Has this been happening already? I have been using this script for about 4 months now, and I had no problems yet. I do syncing once a day.

Oh yes, https://news.ycombinator.com/item?id=20064169

Did they ever post the post mortem? It doesn't sound like they changed policies that automatically block flags accounts.

Yes, it's in that thread a few days later albeit not easy to find.

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

I used to use Tarsnap and am still stuck with it in several projects. I wish I weren't. Restoring backups takes ages. If you need to restore a production server in an emergency, and the whole server remains down for an hour because that's how long Tarsnap takes to restore some tens of Gigabytes, then you will very quickly look for alternatives. I raised this issue with Colin, Tarsnap's creator, multiple times. And the answer has always been "maybe some day". I now simply encrypt backups myself and store them in an S3 or Linode bucket. Much faster. For personal backups, I love restic syncing to Backblaze, where I back up 1.4 TB or so for $5 per month.

$0.25 /GB is pretty expensive. I'm using Hetzner Storage Box [1] which has tonnes of connectivity options where 1TB is only 9.40€ (works out at 0.0094€ /GB)

[1] https://www.hetzner.com/storage/storage-box

I've never heard about Hetzner before, but those are some good prices.

Have they been around for long, and you know of any external public audit that has been done on them?

Hetzner is (one of) the largest German dedicated server providers.

They are DIN ISO/IEC 27001 certified: https://www.hetzner.com/unternehmen/zertifizierung. Of course if you only store encrypted data you don't really need this auditing.

Yeah I've been with them for 8 years as they're the best value hosting provider [1] I've found, which I use for all my memory or computational heavy workloads as they come at a fraction of the cost of what it would cost on AWS.

[1] https://www.hetzner.com/dedicated-rootserver

Hetzner has been around since 1997 (as you will be able to tell from some of their UI).

Yes, Tarsnap is really quite expensive if you have data that doesn’t get deduplicated well.

> I don't get anything for recommending Tarsnap; I'm just a customer. https://www.tarsnap.com/

From their site:

"The Tarsnap service is built on top of the solid platform provided by Amazon Web Services."

I would actually compare DigitalOcean to a Facebook or a Google or an Instagram. You’re banned or blocked and cannot reach a human to find out why or what can be done. DO support is just a bunch of auto mailers sending the same template emails, in my experience.

As tarsnap is run by a single person, it seems hard to suggest it to serious businesses due to the inherent risks there as well as the extremely high storage costs and difficult cleanup procedures compared to competing tools. I tend to prefer solutions like restic + B2 for their price and support, where you at least have a larger company behind the storage backend, you could even directly use it with S3 and it'd still be cheaper than Tarsnap.

Decent write up on the cryptography in restic here: https://blog.filippo.io/restic-cryptography/

Some tarsnap issues detailed here: https://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/

Oh my word, that article on tarsnap is a frightful bore. I was a tarsnap customer and I thought it was great. If Colin is happy running it more like a utility than a rapacious VC-backed hyper-growth racket, I'm not sure why that is bad.

The only reason I stopped being a customer was because another rapacious racket of a business, Comcast, introduced miniscule but enforced data caps in my area, so online backups aren't attractive anymore and I've gone back to external drives and offsite rotation. When I cancelled, Colin sent me a personal e-mail to make sure it was alright to delete my backups. It was probably the best exchange I've ever had with a service provider.

> If Colin is happy running it more like a utility than a rapacious VC-backed hyper-growth racket, I'm not sure why that is bad.

I don't doubt your happiness, but I confess that I'm having trouble reconciling "running it more like a utility" with charging 25¢/GB per month for storage. That is just staggeringly high. What I'm paying $6/month for with Arq would be over $160/month if I were using Tarsnap, and I'm getting end-to-end encryption, deduplication, and versioned file backups. What advantages does Tarsnap bring to the table that justify such a tremendous cost?

The whole point of a backup service is that most of the time you don't need it at all, but when you do need it you really need it. From that perspective, it's like paying for insurance.

I'm not a tarsnap customer, but I think what you're paying for is a service built by a literal obsessive genius that will 100% work when the chips are down.

I wish things turn around and smaller shops make a comeback. Big is not always better, in fact the service is almost always an afterthought.

"But that won't scale!"

Lots of little ones would scale. And lots of jobs. It worked before...

Unfortunately restic was a no go for me due to not being compatible with B2 keys that only have the permissions readFiles,writeFiles,listBuckets,listFiles (no deleteFiles). I don't want the attacker to be able to delete any backups if the manage to get to the B2 keys.

I believe this is the ticket that would add support for this to restic: https://github.com/restic/caddy/issues/2

You can work around this by sticking rclone in the middle which does support soft delete: https://pricey.uk/blog/restic-backups-without-delete/

In short, give rclone your keys, (small ~/.rclone config file) tell restic to use an rclone backend and add an extra argument when running restic. (See the blog link above, all explained)

There's an open issue for this which doesn't seem to have moved: https://github.com/restic/restic/issues/2134 - it does have a couple of PRs linked on there which implement proper soft-delete.

Nice blog post! That's exactly what I needed earlier last year when I was looking at this stuff :)

Looks pretty straightforward. I am surprised it is not more of a priority to get merged though, it is so essential.

Tarsnap seems to have become a multi-man operation.

https://github.com/Tarsnap/tarsnap/commits/master

The other committer is his brother.

Does it matter?

Yes. Having family help out on a 1-man show is an entirely different beast than hiring a full-time employee with wages, taxes, and healthcare. One of these two is a life-long trusted human that you've shared numerous life experiences with, the other is not.

Looks like the brother just does the merges? (which are mostly automatic) so IMO best it matters because it's still mostly a one man, or at least a one family operation - i.e not very resilient to random acts of nature (which backups are supposed to protect from)

Having them be two brothers makes for a lower business continuity risk than having it be a business owner + 1 random employee.

When a business becomes serious? Do you need to hire several workers to look legitimate? If that's still not enough, do you need to make yourself a slave of VC? Only problem I can see is that one man operation is susceptible to the run over by a bus scenario. However that is still better than a woke VC asking the "owner" to cancel some users.

I would seriously prefer a service like this to be run by a two man team. Or at least have a guarantee that I'll get automatically notified if something happens to a sole operator.

Am I reading that right, that 1TB of storage used would be $256 per month + bandwidth costs?

You should read HN posts from the past where @patio recommended that tarsnap should raise prices even more.

I just did the same calculation and I that looks right

I redid it multiple times to be sure too, that's basically 50 times more expensive than other backup options, crazy!

Wait, their pricing is $0.25 per GB per month, so $250 per month for 1 TB? What the heck?

The price is after tarsnap deduces and compresses, which it is quite good at and costs for 1 TB of real world data are much lower.

https://www.tarsnap.com/deduplication-examples

My issue with tarsnap vs Dropbox is entirely workflow. Right now I can treat my Dropbox effectively as an external drive that happens to live on my file system by making everything selectively synced. If I want something backed up, I just drag it into that folder. But once it’s backed up I can turn on selective sync and free up my disk space.

Backup of personal data is often a 1-way endeavor — here’s a dump of photos I scanned that I don’t need to look at anytime soon. But with tarsnap I can’t do that, nor can I then have the ease of browsing to the file and just opening it while it transparently downloads on demand.

If you're transfering data to a backup service and then deleting it locally that's not backup, that's just cloud storage.

It’s the best of both worlds. I can selectively move things off and seamlessly have them move back.

How do you make them sync?

Which “them”?

Damn! Thanks for the info. I don't get these companies. That person should be thanked and not deplatformed.

This seems insanely expensive compared to Restic combined with backblaze

Colin Percival is obviously a smart and very skilled guy but I am mystified why people on HN keep recommending tarsnap, it seems a terrible product for almost any imaginable audience.

If I'm a normal end user, I will probably get a vastly easier to use product at about 1-10% of the price from Backblaze. And if I'm a serious business that can easily afford the > 10x premium and engineering to configure the backup I probably wouldn't want to entrust it to some company with what looks to be a bus factor of one and the apparent technical limitation that a restore might cost me a few DAYS of unanticipated downtime [1].

I mean, sure, if you are some unix nerd wanting to backup your dotfiles and a few small documents (or repos) for maybe around ~$50/year, why not go with some artisanal backup service for HN street cred (and a laudable open source donation policy)? But what other good use cases are there?

Am I missing something obvious?

[1] https://news.ycombinator.com/item?id=25621093

E.g. relative to the author's script, tarsnap get you automatic deduplication across backups... which, if you're like me and basically generate more and more data, allows you to store lots of historic backups at basically no overhead over just storing today's data.

Also, good encryption and security; e.g. Tarsnap is trusted by Stripe (https://www.tarsnap.com/testimonials.html).

Tarsnap does have real downsides - restores can indeed be slow, bare per-GB cost is high, and "like tar" is not a user interface that everyone will like - but there are definitely upsides, too.

That's why so far best idea is to go bare metal with couple of different providers and sync data to multiple servers. In case one provider decides to cancel you, you can still have your data in another place. You can also have a copy at your own server. To have access to all machines as if they were on a local network, you can use tinc-vpn or for convenience ZeroTier.

This is really awesome! I will definitely try this out. Thanks again.

what's the advantage of tarsnap over say, doing an encrypted backup with borg to my gdrive folder? I've never heard of Google random deleting encrypted blobs from people's storage and it's way cheaper

> DigitalOcean does deplatform people... I can suggest Tarsnap

Tarsnap runs on Amazon.