News for hackers. Stuff that matters.
Re:Backblaze IPO by robbiemitchell on Tuesday November 02, 2021 @10:43PM (#29090536)
I already had other (newer) computers being backed up, and this was a courtesy email to tell me I had failed to stop paying for monthly backups for a computer I had not opened (and thus synced) for years. He emailed me because the version running on that computer was considered so unstable as to be unreliable.

Yes, they could have proactively canceled my bill. Maybe they would have! We'll never know, because I responded to this email right away.

He responded to my questions in a lengthy email, which I'll paste again here because I had the foresight to ask in advance for future permission to post publicly.

===

Hi Robbie, thanks for getting back to me! It's hard getting through the email marketing overload. :-)

> Would you be amenable to holding off on deleting the backup > for two weeks (instead of one)

No worries, the most important part is that you are aware the computer is no longer actually performing fresh backups of new data. We can wait a while.

> would you mind if I published your email publicly?

Go ahead, and one of our marketing guys (Yev) already saw your tweet go by, it made everybody in our office laugh too. I'm off in my corner watching logs and emailing the random old customer, they didn't know I do this. :-)

One of the things I've been amazed at doing backup software is how many assumptions I made at the start that aren't true and the MODERN Backblaze software works around:

1) assumption: your disk on your laptop will reliably store bytes or fail. Wrong, it can throw random bits, undetected unless Backblaze keeps its own parallel check sums.

2) assumption: laptop RAM works reliably. Wrong, I think there is a reason servers run ECC RAM, and I'm confused why many laptops choose not to. I guess saving $5 is more important than data Integrity?

3) assumption: the HTTP posts either work or fail. Wrong: WAY MORE OFTEN than the file system are undetected silent data corruption. Solution: client end to end SHA hashes.

4) assumption: customers are sane and won't "edit" the Backblaze internal data structures sitting on the customer's laptop. Nope! If customers are computer beginners they tend to behave nicely, and the most advanced customers are very careful, but there is this crazy middle ground who know enough to be dangerous to themselves. :-)

I really really think that the most modern version of Backblaze (2.3.1 and later) is quite stable and might run flawlessly for a decade on a computer, but some of those early versions like anything starting with a 1.0 or 1.5 scare me because there are too many assumptions in them, not paranoid enough. Yes, the old versions WILL WORK for 99 percent of customers, but at our customer numbers that means every day we have to explain why a restore failed to some poor guy who chose to edit his Backblaze data structures corrupting his own backup and we didn't detect it soon enough.

Our most recent client versions are deeply, profoundly paranoid software that basically assumes your laptop is right smack dab in the middle of melting down, throwing disk and RAM errors as an insane laptop owner not only ignores the warning signs of impending data Armageddon, but fights with Backblaze trying to defeat it from backing up because the customer is worried Backblaze is "wasting his bandwidth". I feel good about this, it's like my whole career was training up to write this incredibly important piece of software.

-- Brian Wilson. CTO Backblaze [email protected]

Re:Backblaze IPO by brianwski on Wednesday November 03, 2021 @12:21AM (#29091077)
Hey Robbie-

I am touched you saved my email and remembered me, and that at the time you didn't respond negatively to the message. Thank you for being a customer.

The only thing I regret is that comment about version number of 2.3.1 and the claim that it would run flawlessly for a decade. Oh geez, please upgrade to 8.0.1, you won't regret it.

-- Brian Wilson, CTO Backblaze, [email protected]

Re:Backblaze IPO by argc on Wednesday November 03, 2021 @01:20AM (#29091349)
This conversation makes me think you care about your users and inspired me to go look at pricing for personal laptop backup, and it appears that linux is not supported. Any plans to support it in the future? EDIT I might have not looked hard enough and there are potentially options out there, I just found https://help.backblaze.com/hc/en-us/articles/217664628-How-d...
Re:Backblaze IPO by teraflop on Wednesday November 03, 2021 @02:07AM (#29091552)
For many years, they were consistently and very diplomatically saying that they had no plans to offer Backblaze Personal Backup to Linux users, because Linux users would use a disproportionately large amount of storage and it wouldn't make sense to offer them a flat-rate alternative to the pay-per-GB B2 option.

e.g. https://www.reddit.com/r/backblaze/comments/jsrqoz/personal_...

It seems that they may be changing their tune, at least as of a few months ago: https://www.reddit.com/r/backblaze/comments/m3xynu/backblaze...

Re:Backblaze IPO by Chilinot on Wednesday November 03, 2021 @05:35AM (#29092575)
As a linux user, i have just setup my own backup solution to backblaze using Rclone and their S3 compatible storage.
Re:Backblaze IPO by tallanvor on Wednesday November 03, 2021 @06:27AM (#29092849)
Which is fine and all, but then you're paying per gb/month. Depending on how much you want to back up, it can cost you a lot more than the personal plan.
Re:Backblaze IPO by Chilinot on Wednesday November 03, 2021 @05:46PM (#29100728)
Eh, im storing about 1.3TB of photos, its costing me around $10/month. That is an acceptable cost IMO.
Re:Backblaze IPO by selestify on Wednesday November 03, 2021 @01:39AM (#29091433)
Is there a reason there’s a short limit on how long my Backblaze password can be? That’s what put me off using it before, because who knows what kind of encryption you guys are using if it can’t handle what my email provider is able to handle, all the while securing more sensitive data ;)
Re:Backblaze IPO by brianwski on Wednesday November 03, 2021 @07:37PM (#29101944)
> Is there a reason there’s a short limit on how long my Backblaze password can be?

The current limit is 50 characters. There are reasons we don't let it get super long like 2,000 characters but we could easily double or triple the maximum length it if it was requested. Is there a number of characters you have in mind? I found a chart here: https://i.imgur.com/trR2u8g.jpg that says a randomly generated 17 character password would take 93 trillion years to crack. The chart doesn't even go up to 50 characters. Backblaze also supports 2-factor authentication via SMS or Google Authenticator codes which personally I would highly recommend to customers also.

> who knows what kind of encryption you guys are using

Backblaze has two product lines: "Backblaze Backup" where we wrote the app that encrypts data on your laptop before uploading into our storage cloud, and "Backblaze B2" where you use any third party tool (or write your own) so the encryption in that case is all under your control. You can see a list of 3rd party tools here: https://www.backblaze.com/b2/integrations.html (make sure you scroll down). There are little pictures of penguins by the software that supports Linux, little pictures of an apple for software that support Macintosh, and a little Window icon for the software that support Windows.

For "Backblaze Backup", we use a well known design where we use symmetric AES-128 to encrypt the file, but each file is encrypted with a different AES-128 key, and that key and an Initialization Vector (IV) is then encrypted by a 2048 bit RSA public/private key. You can read about the encryption flow we use in this blog post: https://www.backblaze.com/blog/how-to-make-strong-encryption...

Re:Backblaze IPO by xoa on Thursday November 04, 2021 @11:33AM (#29108458)
Love Backblaze and it's my go-to backup recommendation, seeing this very late so not sure if you'll see my reply but a few thoughts:

>The current limit is 50 characters. There are reasons we don't let it get super long like 2,000 characters but we could easily double or triple the maximum length it if it was requested. Is there a number of characters you have in mind? I found a chart here: https://i.imgur.com/trR2u8g.jpg that says a randomly generated 17 character password would take 93 trillion years to crack.

Usually the reason people want longer "password" options is because they're using a diceware style pass phrase of 4-6 randomly generated words. It's not so much about ultimate security as many people having trouble memorizing decent fully random passwords. Key stretching and multifactor helps further of course. Longest normal English words are something like 21 characters before getting into niche science terms, so very conservatively 160 characters is probably going to fit even an extremely conservative edge case passphrase. That'd be 8x of some of the very longest words (unlikely to be randomly chosen) and match a 128-bit key which is plenty [0].

>* Backblaze also supports 2-factor authentication via SMS or Google Authenticator codes which personally I would highly recommend to customers also.*

I hope you'll consider adding (via Webauthn online or OS or other toolkits to the application) standalone hardware token support (like Yubikeys or Nitrokeys). Those are enormously more secure, convenient, and also have the bonus of eliminating another set of 3rd party dependencies. If you looked at those in the past and skipped them due to lack of device support worth looking again, every major platform should have coverage at this point.

----

0: Also, rather then thinking in terms of "years to crack" which inherently depends on the processor power thrown at it, might be better off considering in terms of combinations (or entropy). A standard symmetric key at this point is 256 bits, which means 2^256 combinations, but with stretching 128 bits is probably a very very safe seed for the password component. So that could be a "password" of [01] 128 characters long. That'd be roughly equivalent to [0-9] 38 characters long, or alphanumeric cased (26*2+10=62) 22 characters long. Current Oxford English dictionary edition I think has around 170000 words. If we figure a diceware program throws out ultra short, ultra long, and ultra weird words for convenience (whole point is something people can remember) so maybe it's cut down to the 70000 most common words >3 characters, that'd take 8. At a more typical 4-6 words, equivalent to maybe a 17 character cased alphanumeric password.

Re:Backblaze IPO by brianwski on Thursday November 04, 2021 @04:15PM (#29112014)
> ... 4 - 6 words ... very conservatively 160 characters is probably going to fit

Gotcha. I'll file a Jira ticket to see if we can easily and quickly raise it to 200 characters (round number). I think we can handle that easily and safely.

> standalone hardware token support (like Yubikeys or Nitrokeys).

I'll run the idea by a few groups and see what people say. For feature requests NOT related to security, traditionally it's kind of a customer voting system. If enough customers want a certain feature it sorts to the top of the priority list and gets done sooner.

For security things it is obviously different, like if a big exploit like Heartbleed is released in the world we drop everything else and focus on that.

This is sort of somewhere in the middle, it's security related so it gets some additional boost up the priority list for that, but it's also a security feature not all customers would choose to use, so it is still affected by whether our sales and support teams are hearing requests for it.

Re:Backblaze IPO by xoa on Thursday November 04, 2021 @07:40PM (#29113787)
>Gotcha. I'll file a Jira ticket to see if we can easily and quickly raise it to 200 characters (round number). I think we can handle that easily and safely.

Thanks so much for the reply, and yeah that's done-forever territory. Even "just" tripling to 150 as you earlier suggested would almost certainly be plenty, into pathologic edge cases.

>I'll run the idea by a few groups and see what people say. For feature requests NOT related to security, traditionally it's kind of a customer voting system. If enough customers want a certain feature it sorts to the top of the priority list and gets done sooner.

>For security things it is obviously different, like if a big exploit like Heartbleed is released in the world we drop everything else and focus on that.

Sensible way to do it. And yeah I agree this is kind of a hybrid of both. It's not a flaw per se, but it's more than a feature enhancement as there are genuine security implications. If you haven't looked into it, there have been cases of SMS hijacking for example, you can find plenty of discussions on HN alone with a quick "sms hijack" search. It's better than nothing but SMS infra just is fundamentally not very secure. One time tokens with a seed are better, but less convenient or secure then a blackbox that can enforce operator presence requirements or at least completely sequester keys.

Even a couple of years ago it would have been more annoying to integrate but the standards are progressing well enough now that it's probably worth another look. Adoption of hardware keys in business also finally seems to be hitting critical mass, though if we're in an S-curve at last this time around it's well before the steep slope.

Anyway, thanks again and best of luck with everything! As it happens plan to link up a new TrueNAS installation with B2 starting tonight :).

Re:Backblaze IPO by tinus_hn on Wednesday November 03, 2021 @07:17AM (#29093142)
But what about the part where you said you were going to delete their files in a week? I don’t understand why someone would say that.
Re:Backblaze IPO by jmspring on Tuesday November 02, 2021 @11:42PM (#29090894)
I recall chatting with Brian when we started Zetta. It wasn’t a match in terms of tech/direction, but I’ve been a loyal customer as well for years.
Re:Backblaze IPO by throw6622 on Tuesday November 02, 2021 @10:53PM (#29090603)
What a fantastic response!
Re:Backblaze IPO by oh_sigh on Tuesday November 02, 2021 @11:12PM (#29090724)
Geez, this guy really is a genius.