I'm looking into supporting Glacier in Arq. It sure is cheap -- $10/month for a terabyte.

Arq is great backup software. I would view inclusion of Glacier support as a worthy paid upgrade.

Arq is the pinnacle of my rather large backup pyramid which also includes: Dropbox, Superduper, Crashplan, rsync, SVN/GIT and more.

In an earlier version of Arq, there was no possibility to see which data was actually selected for backup (and which not). Has Arq become more user-friendly?

I'm a current user of ARQ, and it has, in my opinion (compared to Backblaze) a phenomenally user friendly description of exactly what is being backed up, and when, and when something was added, or modified.

I think the key here is not to just provide a toggle for using Glacier instead of S3, but to have the historical snapshots migrated to Glacier from S3 and deleted every 90+ days.

I will gladly pay more money for Arq again.

I think it would be more like choosing Glacier instead of S3 on a per-folder basis.

Arq is incremental backup. The data structure is similar to git's http://www.haystacksoftware.com/arq/s3_data_format.txt

Unless your files change a lot, keeping the latest backup version in S3 and previous versions in Glacier would mean that most of your backup data are still in S3 I think. Right?

Yes. The forthcoming S3 Lifecycle Policy API additions should allow for exactly this, automatically.

Just FYI, I too would view Glacier support as something worth a normal upgrade fee. I am using Arq more as a long-term back-it-up-and-forget-about-it storage solution anyway, so it seems like a natural direction to go for a user like me (same strategy, but even lower cost).

That said, I am reminded that I should not forget about the Arq backups and do a few test restores sometime. :)

As long as you promise to have the glacier update be a free one, I am buying an Arq license this evening the moment you tell me it will be so! :)

As a happy customer, I'm glad to hear this as well!

That would be fantastic.

That's Mac-only, right? Can anyone suggest an equivalent Windows tool for backing up to Amazon?

Jungledisk, Duplicaty (LGPL)

That would be Duplicati -- my favorite backup utility.

There's Cloudberry Backup. I haven't used it myself, and I'm told it's not versioned backup, but it backs up to your own AWS account.

Wow, yes, please do! I've never used Arq, and at the S3 prices it doesn't make sense on top of my Dropbox. But with Glacier, I'd love to be able to back up my entire HDD off-site for $5/month, and Arq looks like a good way to do that.

I will start using Arq when it supports Glacier. When will it be ready ;)

I'm a happy Arq customer today and I'd pay for an upgrade to Glacier! Longer term I'm sure you will have competitors on Glacier that you will be competing with so best to move sooner than later.

Please!!! I'm a very happy Arq customer, this update would be awesome!

Arq, please add support for Glacier. - Happy customer.

I'll buy Arq immediately when Glacier gets added

Me too. If it was not for the recurring cost of S3, I would have bought ARG already. It's brilliant software.

Yes please!

Do IT! Fred

With large volumes the real issue is not the storage but the upload speed. I've done some experiments and with the Comcast link (20Mbps down/whatever up), I got 1 Gb/hour upload rate. So, it'll take 11 full days to upload 256 Gb. Or, more realistically, if you do it overnight (8 hours) - the entire month.

If you don't mind the additional cost, you can physically send Amazon a harddrive: http://aws.amazon.com/importexport/

nothing says "cloud" like sending a physical hard drive through the mail.

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S." - http://en.wikipedia.org/wiki/Sneakernet

There's also Amazon Direct Connection (http://aws.amazon.com/directconnect/) if you happen to be on the same backbone / data center.

But then you can do incremental backups which should be faster. Although I admit I am not sure how to do it with encryption (impossible?).

That's what duplicity is for: http://duplicity.nongnu.org/

It uses rsync's algorithm to figure out what has changed, and everything is encrypted locally.

(The site says it's in beta, but it's said that for years.)

Makes me wonder what Amazon is doing to commoditize its complements, i.e. what it's doing to improve high speed internet access. There are a lot of people in my vicinity who have no option faster than a wireless 3Mbps connection capped at a few GB per day. And this is within easy distance of Amazon's East Coast data center.

Amazon should be big fans of Google Fiber. With a gigabit this stuff is a lot more impactful.

Home-use is probably the only situation Glacier is good for backup though.

A home user is fine with a 3.5-4 hour window before their backup becomes available for download (as it will probably take them days to download it anyway).

In a corporate environment, I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation.

Seems good for archive-only in a corporate environment (as the name implies).

Interesting to note - those of us who used Iron Mountain/Data Safe for years - 4 Hours was considered a "Premier Rapid Recovery" service that we paid a lot of money for (as recently as 2003, actually).

In a true disaster recovery (Building burned down or is otherwise unavailable) - it usually takes most businesses a week or so just to find new office facilities.

But - agreed, there will be some customers for whom Glacier wouldn't work well for all use cases.

Now - a blended S3/Glacier offering might be very attractive.

Yes, that's what I am thinking: a week of backups to S3 and a script that does moves the oldest one from S3 to Glacier. You would rarely need a backup older than a week if you've already got a week of backups. I'm still figuring out if there is a practical way to do this with incremental backups without introducing to much risk that the backup process gets messed up.

nice!

It's made for archiving, not 'backups'. You're expected to keep most recent backups on-site, so if a RAID array dies or you happen to do something wrong, it just takes minutes to restore. Off-site, or tape archive, it's usually another thing. Most of the data is never, ever accessed (but required by company policy to be kept).

In a corporate environment, I wouldn't want to depend on the cloud as my primary backup solution in the first place. I'd be much more comfortable using it as the offsite mirror of an onsite backup. If you're at a point in disaster recovery where you have to restore from your offsite, you (likely) have bigger problems than a 4-hour wait time.

I personally believe that data should never be deleted (or overwritten), but only appended to. Kinda like what redis/datomic does. So, keep live data onsite, along with an onsite (small) backup, and all the old data in Glacier.

You can believe that, but legal realities dictate otherwise. There are certain classes of information that you are not permitted to keep beyond a defined horizon, either temporal or event-based. Legal compliance with records management processes means having the ability to delete or destroy information such that it cannot be recovered. Note that if the information is encrypted, you can just delete the decryption key and it is effectively deleted.

That's just the beginning - if you've ever been in a ediscovery process, having large amounts of historical data is actually a liability - if instead of 100GB you have 10TB, you'll need to hand that over, and before that, to cull it so you don't inadvertently hand the opposition a huge lever to be used against you. Processing and reviewing 10-100x the data can take inordinate amounts of time more than you expect.

> Note that if the information is encrypted, you can just delete the decryption key and it is effectively deleted.

Does this satisfy the legal requirements you mention? I assume there are further requirements on what constitutes sufficient encryption?

"In a corporate environment, I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation."

I am guessing you work in a company with a good IT department then, I am guessing this is not the average. Many companies I have worked for 4 hours would be a miracle with a 1-3 day operation minimum.

And don't forget that the X00MB type size limits many IT departments puts everywhere it not because getting a TB hard drive is cheap, but because all of the extra backups add to the cost of each new MB. Having another extremely cheap way they could backup large amounts of data (encrypted?) would help to reduce the cost of each extra GB.

I don't think DR is the correct use-case. I think it more likely to be (as in the case of banks which I know best) a regulator asking for electronic documents or emails relating to a transaction from 6 years ago (seven years is the retention requirement).

In that case, 3-4 hours would be more than acceptable.

Here, we are producing logs of simulation runs that may be needed in the next couple of years. Most of the logs will be destroyed unused. This is a perfect use case for glacier. We do not care about 5 hours recovery time.

Absolutely, it's an archive product, not a backup product.

In a corporate environment where you have low RTO goals for DR scenarios, relying on backups instead of replicated SAN etc is not a sound practice, especially with large data sets.

"I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation"

ding!

This would work great as a disaster recovery solution for the museum I work at. Not just for home-use.

What's even more important, you will be able to encrypt your backups without having to disclose the encryption key in case you ever need to restore (client-side encryption and decryption). This is not the case with Backblaze, which is why I switched to CrashPlan — but I'm still looking for other solutions.

CrashPlan shows your encryption key in plain text. Doesn't that worry you?

You'd prefer that they lie to you and pretend it's not stored in the clear? If you don't have to type it in every time a backup runs, it's in the clear - everything else is just window dressing.

Standard practice is IMHO to use the user or system keychain.

Right, which is a largely meaningless distinction: if the process runs as you, any successful attacker can simply read it out of the CrashPlan process in memory.

Again, if you're not typing the password in every time, a local compromise is almost certainly game-over. Apple's keychain helps reduce the damage if the data's not actively used but for something like CrashPlan which is always running the attacker is probably going to be lucky.

I agree that showing the encryption key in the clear is not a serious security flaw but it's IMHO against best practice.

Passwords and key are usually shown in an obscured form, usually with asterisks, and stored in the user or system keychain. You are absolutely right that the security value of these standard practices should not be overvalued but still … what else within the security framework of CrashPlan is not done in accordance with best practice?

(I assume, BTW, that CrashPlan does not use the system or user keychain on the Mac because it is not a real Mac citizen but a Java-based app. Firefox and Wuala – the latter Java-based too – don't use the user or system keychain either.)

There have been sound arguments made against obscuring password entry.

Those arguments are typically made from the wrong perspective. It's just so common that if you don't do it, your product will be perceived as insecure. And as long as actual security is not horrible, the perception of security is what drives sales, not the actual security.

CrashPlan defaults to using the account password as the encryption password.

However, you can also secure the encryption with a password not associated with the account. Or even provide your own 448-bit key. If you do either of these options, CrashPlan support will not be able to help you

This setup allows CrashPlan to easily help non-technical home users, while allowing technically savvy users to securely hang themselves with their own encryption.

My question is not about the setup, that's OK. I am wondering why CrashPlan shows the encryption key in the clear and does not store it in the user or system keychain.

As others have noted, it is partially because Crashplan is a Java based app. Partially, it is also because Crashplan runs as System, not as the user. That way, I can have Crashplan backup my wife's user account and my user account.

Furthermore, you can use the encryption key + a custom password, or your own encryption key with a passphrase. In this case, it is encyrpted locally and the key is not sent to Crashplan[1].

[1] http://support.crashplan.com/doku.php/recipe/change_security...

I'm assuming this because their desktop client is a java app and it it's been made to run on linux, mac and Windows. Is there such a thing as a keychain on Windows? I've only used it on linux and mac.

Yes, there's the CredWrite API - http://msdn.microsoft.com/en-us/library/windows/desktop/aa37..., although I must admit I like the OS X keychain a lot better.

I see what you mean. I have up voted your original comment.

Where does it show that? And are you sure we are talking about the encryption key, not the account password?

Settings / Security / Archive Encryption / 448-bit encryption with custom 448-bit key

Then no, it does not worry me. Obscuring password entry in an application that I almost never run is not a problem. Plus, if I do actually run the app and want to enter the password, I don't do it with a bunch of people peering over my shoulder.

The other feature that's great about CrashPlan is that it will allow you to backup to other systems running their software. I backup my laptop to both their service and one of my dedicated servers. That way if anything ever happens to their online service I have a second remote copy that I control.

Backblaze does support the optional use of a separate encryption key which they claim to never store, but there's no information about whether it is ever disclosed to them or not. There's little information on their website, but it can be set in Settings > Security in the Backblaze client.

You have to provide your encryption key to their website if you want to recover files. That's a show-stopper for me.

A possible alternative to Glacier+product X (you mention Arq) could perhaps be Git-Annex, recently mentioned as it's Kickstarting a friendly UI component: http://news.ycombinator.com/item?id=4385886 http://www.kickstarter.com/projects/joeyh/git-annex-assistan...

Already has Amazon support: http://git-annex.branchable.com/tips/using_Amazon_S3/

Very friendly with very large files!

Here are Arq developer Stefan Reitshamer's initial comments on Amazon Glacier:

http://www.haystacksoftware.com/support/arqforum/topic.php?i...

I use s3cmd sync to keep my files backed up. But for my pictures, tunes, I use a Linux server and rsync. Now glacier is perfect for my tunes and family pictures.