News for hackers. Stuff that matters.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @11:31AM (#14139666)
We have always tried to squeeze costs out of storage. That required a ton of tech/ops work. Bandwidth it turns out isn't that expensive - seems to just typically be overpriced. Hopefully lowering the price will enable people to do more with their data.

I'd love to hear what you haven't done in the past due to bandwidth/download pricing that you'd have wanted to...?

Gleb @ Backblaze

Re:Backblaze B2 Drops Download Pricing by 60% by Veratyr on Tuesday April 18, 2017 @01:25PM (#14140825)
I'd be interested to know, have you looked into reducing/eliminating bandwidth prices further for clients close to your network? It seems like you're with Unwired, which has heavy inbound traffic and peers at a number of facilities in the Bay Area. It'd be really interesting to know if you could, for example, offer free traffic to Linode, DigitalOcean or Vultr, all of which have offerings in the area and could help balance out Unwired's traffic flow (I think that's a thing ISPs care about).

The big thing missing from B2 for me now, relative to the other big players, is the ability to process my data once it's stored with you. If I store my images in S3 for example, I can use EC2 to thumbnail them without paying anything for retrieval. At the moment I can't do that with B2 and it seems unlikely to make sense for you to build your own compute cloud but perhaps you can integrate with others.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:35PM (#14140945)
I would love to do this. Cross-connects aren't free, but they are certainly less expensive than going out across the Internet. And enabling compute/storage use cases by (possibly) partnering with other providers would be excellent. I'll reach out to a few of those compute providers and explore.

Can you tell me more about how you use/would use cloud storage? (with or without compute)

Re:Backblaze B2 Drops Download Pricing by 60% by _mgu7 on Tuesday April 18, 2017 @01:56PM (#14141164)
Currently I don't really use cloud storage, it's cheaper for me to colocate a 2U stuffed with disks and buy transit.

Primarily though I store media. I have (for a home user) large amounts of high resolution (lossless audio, RAW images, high bitrate (even ProRes) video) media that I like to keep for archival and transcoding but don't need for playback. I want to store the original media somewhere safe and cheap but have the ability to transcode it to a format suitable for playback/streaming on demand.

Actually it just occurred to me that it might be worth you talking to the folks at Plex (https://plex.tv) about this too, as they're working on their Cloud product and need more places for their users to store media.

Re:Backblaze B2 Drops Download Pricing by 60% by jjeaff on Tuesday April 18, 2017 @02:59PM (#14141804)
If I was a storage company, I would consider Plex a hot potato and would stay far, far away. There is a reason Plex isn't building out their own storage and the reason isn't cost. It's because they know that 99% of all the media their customers handle is pirated content. At some point, Plex will become large enough to get on the radar of the MPAA and at that point, I assume they will go after the cloud storage providers.
Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @04:20PM (#14142508)
Yeah, I'm not sure about that. Plex has nothing to do with how users are getting the content, it just offers playback and cataloging after they've already retrieved it.

I'd agree if they allowed something like library sharing between users, but I don't see them as a likely target just for allowing playback of consumer media.

I use a (local) Plex installation for home videos, downloaded clips, and various other non-infringing things. It's the most convenient way to share the library out to a Chromecast/phone/Roku.

Re:Backblaze B2 Drops Download Pricing by 60% by jjeaff on Tuesday April 18, 2017 @09:57PM (#14144868)
Ya, I don't think Plex would be necessarily at fault, although they could be susceptible to DMCA takedown notices. But a storage provider could get into deep trouble if they know tons of pirated content was being uploaded. Kim Dotcom claims MEGA had no idea if the content being uploaded was infringing.
Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Wednesday April 19, 2017 @12:15PM (#14148883)
Megaupload (which I assume you mean because afaik MEGA has not been in legal trouble) was a distribution platform; the purpose was to upload something and give the link to others. If it was a personal file locker that couldn't link out or be shared, as Plex intends to be, it probably would not have had issues.
Re:Backblaze B2 Drops Download Pricing by 60% by Veratyr on Tuesday April 18, 2017 @03:06PM (#14141876)
I feel like it'd be the opposite. DMCA protects the storage provider as long as they're unaware that they're storing infringing content and Plex limits sharing enough that any infringement is unlikely to reach the MPAA/RIAA.
Re:Backblaze B2 Drops Download Pricing by 60% by jjeaff on Tuesday April 18, 2017 @09:55PM (#14144862)
I think inking a deal with Plex would be pretty obvious. I'm not sure where the line is for knowing that you are storing infringing content, but MEGA claims they didn't know what users were uploading.
Re:Backblaze B2 Drops Download Pricing by 60% by Veratyr on Wednesday April 19, 2017 @12:54PM (#14149266)
Plex itself is a US company and were it "obvious" that its existing Cloud product was infringing on copyrights, it would have already been the target of a lawsuit.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @02:42PM (#14141617)
Got it. Transcoding & making image thumbnails are two common use cases we hear. Thanks.
Re:Backblaze B2 Drops Download Pricing by 60% by derekp7 on Tuesday April 18, 2017 @02:53PM (#14141744)
I can give you my example -- I have an open source backup tool called Snebu which uses an sqlite DB to store the backup catalog (file names, file hash, metadata etc), whereas the individual files are stored in a vault area (currently just a disk volume, with files named after the sha1 hash). Backup data is sent to it by piping in tar format to the main process, which updates the catalog DB and deposits files into the storage vault.

I'd like to add cloud capabilities, but for that I'd have to run the backend process somewhere that can easily update the DB, and easily read/write the data in the vault, along with syncing the full DB to the vault location after a backup.

My other option is to run everything locally, and compute a delta file between versions of the DB and send that delta over to the vault (i.e., do D2D2C backups).

Edit: Another possible requirement -- I'd like to pack multiple smaller files into an archive file, and deposit that into a bucket. But as files age out of the archive, I'd like to be able to move active ones out of it and into a new one. Right now, I'd have to re-create the still-valid backup data out of on-hand information, upload the new archive, then send a command to delete the old one.

Re:Backblaze B2 Drops Download Pricing by 60% by loxias on Tuesday April 18, 2017 @03:40PM (#14142186)
Speaking for myself, I'm excited about using B2 as a replacement for more expensive local disk in not-heavy-iops-role machines. It's not the most exotic use, but if you look at every vm provider (linode, digital ocean, even amazon) the prices of local disk are huge in comparison to the prices of cloud storage like B2 & S3.

By combining them I can have a cheap box (Linode for instance) with a local write cache, with B2 providing a terabyte of raw storage, and get a slow but effective VM with a 1TB "local disk" for under $20 month. Which is great.

Re:Backblaze B2 Drops Download Pricing by 60% by Veratyr on Tuesday April 18, 2017 @03:51PM (#14142278)
Unless you're going to be doing pretty much entirely sequential reads of entire files, I don't think this is likely to work well for you. B2 (and other cloud storage) isn't just priced by bandwidth but by "transactions" (requests).
Re:Backblaze B2 Drops Download Pricing by 60% by loxias on Tuesday April 18, 2017 @04:04PM (#14142373)
It's true, there is a price for some API requests, but it's actually quite low if you're using the cloud storage to populate a local cache! Backblaze categorizes download as a "class B" transaction with 2500 free per day and $0.004/10k. I fully admit it's not a suitable solution for many workloads, but I've been playing with a VM using B2 as slow additional disk (as an additional volume, not suggesting putting / or /usr..) and am yet to exceed the free tier.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @06:51PM (#14143751)
So are you actually running a VM image somewhere and using B2 as a virtual disk drive from it?
Re:Backblaze B2 Drops Download Pricing by 60% by loxias on Tuesday April 18, 2017 @02:56PM (#14141771)
I would love this! Great idea. So far, B2 beats S3 on nearly every metric, with the exception that with S3 I can process data more cheaply with EC2.

A crosspromotion deal between Linode and Backblaze would be fantastic.

Re:Backblaze B2 Drops Download Pricing by 60% by infogulch on Tuesday April 18, 2017 @01:38PM (#14140972)
So you're asking for cheaper egress to specific cloud provider peering partners that offer compute. ?
Re:Backblaze B2 Drops Download Pricing by 60% by Veratyr on Tuesday April 18, 2017 @03:32PM (#14142110)
It doesn't need to be a specific provider, those were just examples. The idea is to have free traffic between Backblaze and someone that provides compute, so you can transform the data without fetching all of it. I suggested those in particular because I believe they're all in the SF bay area and should be present at one of Unwired's existing peering points.

The reason I can't see traffic just being straight free for everyone is that for Backblaze to make traffic free, it needs to make arrangements with the providers to reduce costs (for example by connecting directly rather than through the internet).

Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @01:28PM (#14140859)
B2 is a competitor of long-term storage solutions like Glacier or TarSnap, not virtualized storage like S3. It's meant to be almost 100% write-only. Basically, you pull your backups down in case of disaster.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:37PM (#14140963)
Backblaze has a backup service for laptops/desktops, but our B2 service is actually more like S3 - the data is available without delays or penalties for access. (Unlike Glacier.)
Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @01:48PM (#14141086)
Yeah, it's similar to Google Nearline or TarSnap in that unlike Glacier, the data can be read out instantly instead of waiting for Amazon to restore it off of a tape or whatever they do. The "penalty for access" is the download cost.

However, my understanding is that B2 was intended for long-term storage and not everyday cloud storage. Is this understanding incorrect? Backblaze does take the position that B2 should be considered as a competitor for S3?

Up to this point, my experience with B2 has been that they don't want to be the authoritative storage source for the data. I seem to recall reading that B2 should not be used as primary storage (though that was in beta days). In support discussions, a representative expressed surprise when it was implied that I may not have local copies of files that already exist on B2 (I do indeed have local copies, but it was beside the point) before confirming that yes, I would need to redownload and reupload to complete a rename.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @02:37PM (#14141567)
Brian from Backblaze here.

> my understanding is that B2 was intended for long-term storage...

Long term storage is a great use of B2, but it was never "intended" as anything other than the ability to reliably store blobs of data and retrieve them really fast whenever and how ever often you like. I suppose there are different use cases that work well for B2 and others that don't work well, but that is up for customers to decide, not Backblaze.

> ... they [Backblaze?] don't want to be the authoritative storage source for the data

Who told you that? Backblaze wants people to use B2 for whatever they want. Backblaze makes an AMAZINGLY GOOD authoritative storage source.

> I would need to redownload and reupload to complete a rename

Right now the B2 service does not include a "rename" function. If that would be useful to customers, we can add it? We collect customer requests and add them in the order that we have time and what gets the most customer votes.

Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @03:04PM (#14141849)
Thanks for clarifying.

>Long term storage is a great use of B2, but it was never "intended" as anything other than the ability to reliably store blobs of data and retrieve them really fast whenever and how ever often you like. I suppose there are different use cases that work well for B2 and others that don't work well, but that is up for customers to decide, not Backblaze.

I guess that the pricing structure of free uploads and expensive downloads is the main thing that caused it to be compared to Glacier and other long-term storage solutions like Nearline. Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?

>Who told you that? Backblaze wants people to use B2 for whatever they want. Backblaze makes an AMAZINGLY GOOD authoritative storage source.

Like I said, I must be recalling discussion from the time the beta was announced, and that such restrictions were shed post-beta. Did Backblaze ever expand into its promised second datacenter?

>Right now the B2 service does not include a "rename" function. If that would be useful to customers, we can add it? We collect customer requests and add them in the order that we have time and what gets the most customer votes.

Yes, this made me stop using B2 for all practical intents and purposes. It's such an easy thing that it doesn't seem like there's any reason it shouldn't be supported. Between the web UI's lack of pagination and the inability to rename a file to comply with virtual folders, the web interface is not usable.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @06:04PM (#14143406)
> Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?

A price drop MIGHT open new markets to us. What those markets are are kind of an internal debate at Backblaze. What we agree on is that we priced outbound bandwidth incorrectly when we launched B2. We know this for two reasons:

A) The price Backblaze pays bandwidth is WAAAAAY lower than 5 cents per GByte.

B) Almost nobody is choosing to use Backblaze as a CDN or website server or a large source of outbound data.

One new market might be making B2 more attractive as a CDN or website server. It is my understanding that CDN customers use the CDN because of two things: 1) cheap download prices and 2) fault tolerance and reliability and distribution all over the world.

Most CDNs charge about 2 cents/GByte or maybe a little more.

If B2 outbound bandwidth prices drop below what CDNs charge, then SOME of their customers might consider B2 as one part of their solution.

In my humble opinion we are still overpriced, but the reason we're dropping prices slowly is because we're slightly terrified of becoming too attractive too fast and not being able to handle the onslaught as the world abandons all the other CDNs and goes with B2 since we are so much cheaper. So we're going slowly at first while we ramp up, and then we can drop prices much more in the future as we gain comfort with serving up gigantic amounts of cached data.

> Did Backblaze ever expand into its promised second datacenter?

In the process of. Servers are sitting in the new datacenter powered up and talking with each other (and even have customer data on them), but we do not yet let customers choose which datacenter their data lands in (or choose to duplicate their data between).

Re:Backblaze B2 Drops Download Pricing by 60% by extra88 on Tuesday April 18, 2017 @04:46PM (#14142752)
> Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?

B2's download cost was already cheaper than AWS for everyone but the largest customers but I expect a little guy has to try harder.

I think something that has contributed to seeing B2 as mainly for backup, aside from it being a service of a backup company, is greater emphasis on durability rather than availability. And looking at web pages now, B2's availability SLA is 99.9% [0] while S3's is 99.99% [1] (S3 Infrequent Access is 99.9%).

[0] https://www.backblaze.com/b2/cloud-storage.html

[1] https://aws.amazon.com/s3/

Re:Backblaze B2 Drops Download Pricing by 60% by adventured on Tuesday April 18, 2017 @12:41PM (#14140331)
I'd like to permanently, easily offload all image storage & delivery (consumer oriented services) - without being mangled by the obscene bandwidth costs of AWS. I don't want to roll my own solution to get cheap bandwidth, I want a dependable service that hosts files (images in this case) & can serve them up, and do so at very cheap bandwidth prices.

B2 references Vintage Aerial as a success case for storing their large image collection. So I checked out their site, pretty cool service. So I right click -> view image, on the large & thumbnail images on their site and they're on AWS of course (archival storage on B2 I assume).

Hey Backblaze, AWS's bandwidth margin is your opportunity.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwawok on Tuesday April 18, 2017 @01:03PM (#14140557)
Its hard to be highly available, redundant, and cheap.

AWS picked the first two. B2 picked the second two. I don't think they would make a great CDN host.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @06:36PM (#14143646)
Brian from Backblaze here.

> AWS picked highly available, redundant > Backblaze picked redundant, and cheap

Both AWS and Backblaze have had (and will have more) outages. No single vendor is a good choice for a service that absolutely positively must stay up no matter what.

We liked this article that shows how to configure Amazon S3, Microsoft Azure, or Backblaze B2 so you are not susceptible to Amazon outages: https://cloudrail.com/avoid-next-aws-s3-outage-failover-scen...

Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @01:47PM (#14141065)
An OT question about the list() function, which also applies to S3 and Google. Why does it return such a verbose, heavy response. Here's an example: https://www.backblaze.com/b2/docs/b2_list_file_names.html.

If I have a 10000 files, this will be a LOT of bandwidth. I JUST want the list of file names, NOT the other redundant fat. Solution:

1) let me specify which fields I want OR 2) compress the response

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @02:26PM (#14141462)
Brian from Backblaze here.

Backblaze does not charge anything (zero) for the bandwidth of the b2_list_file_names response. What we mean for the bandwidth is "file contents you download". So compressing or not compressing b2_list_file_names won't make any difference in your bill.

We do change a TINY "per transaction" charge on api calls like b2_list_file_names. The reason is to break even on the farm of servers (we call them "API Servers" internally) that are needed to support the HTTPS load generated from those calls. But that is really really tiny, like you get 1,000 calls for 4/10ths of one penny.

Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @02:34PM (#14141536)
Thanks Brian.

I like B2. And thanks for "absorbing" the cost of the list() function.

My sincere question to you is:

Sending this much redundant data in the response can cost YOU a lot, it is not "trivial" when you add millions of requests. How hard is it to allow the user to specify which fields he wants in the response?

Also, please answer my other question about the authorize() token headache.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Thursday April 20, 2017 @01:42AM (#14154246)
Catching up on answers:

> How hard is it to allow the user to specify which fields he wants in the response?

Pretty easy, but like all things we prioritize features based on customer requests. Seriously, you could probably game it by creating 20 different gmail addresses and emailing us once every three days from a new email address requesting the same feature. :-) If we hear from something like 20 different customers the same request we prioritize it higher and get it done. The OTHER way is if a "potentially large" customer approaches us and says they require some feature or they cannot integrate or won't use B2. I won't name the customer for privacy reasons, but we added the ability to put the SHA1 at the END of the upload instead of at the start in the HTTP headers because one "important" customer required it. It also makes sense, the headers come FIRST and some streaming applications don't have the SHA1 until all the data streams through them, thus at the end of the stream.

> redundant data in the response can cost YOU a lot

Much to my surprise (totally being honest), bandwidth only costs Backblaze about 2% or 3% of our total operating budget (including salaries and datacenter). Modern bandwidth is CHEAP! But even more interesting is that most of our Online Backup customers and B2 customers pretty much only UPLOAD data, but we are forced to purchase symmetric bandwidth (same upload as download). So hilariously enough, outbound bandwidth is utterly free to us until it comes up to match the inbound bandwidth. Free. I've even tried to come up with a scheme where we make it free to our customers up until the moment we actually have to increase our capacity to support outbound, but I can't figure out a scheme to make that happen. So we are pricing it for the situation when we exceed the "shadow", even though it is nowhere near (and probably never will) exceed the shadow.

> please answer my other question about the authorize() token headache

I looked back and found one question you asked about 24 hours vs weeks vs forever. If that is the question, the 24 hours is a "guideline" because so many customers kept asking what to expect so we made up a number. In reality, you have to follow the protocol which is: if a call is ever rejected for whatever reason with a "expired_auth_token" or "bad_auth_token" then your code should re-authorize. It's source code that you have to write and it doesn't matter how often it is triggered, after you write it (correctly) then it will always occur when it needs to occur, and never occur when it doesn't need to occur. To give you an insight as to why, if we happen to reboot a particular server (or it crashes, or the power supply dies), then all the auth codes will be incorrect and all currently connected customers that used that authentication server need to re-authorize which will come from one of the OTHER authentication servers. We do weekly code updates on Thursdays at 2pm-ish so if you get an authentication token on Friday it will probably work for 6 days, but if you get one Thursday morning it might fail once in the middle of the day. This is all about cost reduction for Backblaze - we simply don't purchase ANY high end load balancers, it is all cheap computers that are expected to fail and break, but because there are hundreds (or thousands of them), even if half of them have lost power or died there are still plenty up and running (now we have some in a different datacenter!) to respond to your software's requests. We don't own a single load balancer (like F5), and we pass the savings on to you. Unfortunately we also pass along a small amount of additional complexity.

So it is really straight-forward code and you write it once and it just executes every time it needs to and never executes when it doesn't need to. Your code should not have ANY assumptions of time in it. Unfortunately it isn't something where you can write really bad software and just ignore it, you MUST handle the return value EVERY time it is returned, whenever one of our authentication servers has a hardware failure (which is at unknown times) or whenever we restart Apache Tomcat on the authentication servers (Thursdays at 2:15pm-ish-kinda).

Re:Backblaze B2 Drops Download Pricing by 60% by zimbatm on Tuesday April 18, 2017 @03:13PM (#14141946)
Also related to that. In a situation where:

* a bucket has 1M files in it

* uploads are generally of 200 files at a time

* each file name is guaranteed to map to a unique content (content-addressable)

* some of the files to upload might already be in the bucket

It's actually quite expensive to list all of the bucket files to find out if they haven't been uploaded. Especially when the file names are hashes it's hard to filter by prefix as well. (same issue on S3)

It would be great if there was a way to just upload the files and get an early 303 response if the entity already exists and has the same SHA1. That way the client can just push all the files without having to build the "new files" list.

Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @01:17PM (#14140734)
Hi Gleb!

The thing I haven't done due to the download pricing is back up very much data into B2. Since the B2 API doesn't support renames, you have to download and then re-upload a file to rename it. I would be storing a lot more there if renames were available! [Note: they may be available now; I haven't checked for a few months. Please tell me they are.]

I was an early adopter of B2 and uploaded a bunch of files with naming that's incompatible with the "virtual folders" of the web UI (wrote my own script to handle this since Duplicity et al did not yet have support). Now I'm stuck and the only solution offered is just "pay us to download everything again".

Also, the web interface does not have any pagination. Since virtual folders are based on naming, and since rename support is not available, this can flood the browser. I have a bucket that crashes the tab every time I try to open because it's not paged. I could write a little Greasemonkey injector to page it myself, but meh, I don't really care enough to do that. Instead, I just don't really use B2.

When I discussed these issues with support and suggested that at least pagination on the web interface was a basic thing that engineering should be able to accomplish quickly, they did not seem to agree.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:52PM (#14141116)
Interesting. "rename" is generally not supported by object storage, but an API to "copy with a new name & delete" often is, which allows the same result. I presume that would work? (We're planning to add that in the future.) Pagination is on the list as well...
Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @04:26PM (#14142577)
Both of these things seem like such weird things to avoid adding, especially pagination. I can't be the only user to have a bad experience with the web UI because of this. How hard is it to add a simple pager? It doesn't have to be fancy.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @07:01PM (#14143817)
Neither is hard. Want to do both. Just have a list of hundreds of requests and have to prioritize ;-)
Re:Backblaze B2 Drops Download Pricing by 60% by nbarbettini on Tuesday April 18, 2017 @12:23PM (#14140124)
No feedback from me at the moment, other than to say that what y'all are doing with B2 is awesome. It's great to see more competition in this space, and I love the open-source ethos you've cultivated at Backblaze.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:38PM (#14140967)
Thanks for the kudos!
Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @01:05PM (#14140577)
Hi Gleb,

I want to use your cloud storage instead of S3. My main reason for using S3 is the DURABILITY (not reliability). S3 offers 99.999999%, which means it is replicated numerous times. Which means I DON'T HAVE TO WORRY ABOUT taking extra customer backups, it is "baked" into S3.

How is the durability for B2?

Re:Backblaze B2 Drops Download Pricing by 60% by olavgg on Tuesday April 18, 2017 @01:16PM (#14140717)
Backblaze use Reed Solomon / Erasure Coding for their data, plus checksumming with healing, which means that it should be very safe.

Their main disadvantage is that they only have one datacenter, unless they have gotten a second one recently.

Re:Backblaze B2 Drops Download Pricing by 60% by mjevans on Tuesday April 18, 2017 @01:34PM (#14140933)
I was suitable impressed with their description of the singular (I believe) data center they own. Even though it is in an area that is at risk for a major earthquake it seems to be at the upper end of engineering specifications.

Having said that, if your data is absolutely critical then I agree that even further geographic diversity would be desirable. Most likely you want to avoid the Pacific Rim, and volatile countries. A central European site/company would be good for this. Obviously you should expect to pay more to store your data in more than one place.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @06:15PM (#14143497)
Brian from Backblaze here so I'm biased. :-)

> if your data is absolutely critical then I agree that even further geographic diversity would be desirable

I think geographic distribution is absolutely critical if you need "uptime". But for pure durability I think multiple geographic regions is a red herring. What you need much more than multiple geographic regions is having your data stored in two profoundly different formats with two profoundly different vendors with source code written by two separate teams. Furthermore, I would pay for those with two separate credit cards.

The same software bug (or lack of payment resulting in Backblaze B2 or Amazon S3 deleting ALL copies of your data) will delete your data flawlessly across geographic regions. Every time.

Backblaze carefully chose our main datacenter to not be in an earthquake zone, and on a hill (flood proof), and not in tornado zone, and no hurricanes. I suppose that in the case of a meteor strike on a datacenter or terrorist attack some geographic regions would help, but that is really diminishing returns compared with software or human error.

If you truly value your data, two regions won't save you -> two vendors will.

Re:Backblaze B2 Drops Download Pricing by 60% by dyu- on Tuesday April 18, 2017 @09:15PM (#14144652)
> If you truly value your data, two regions won't save you -> two vendors will.

Never thought of that. It is costly but still, great advice!

Re:Backblaze B2 Drops Download Pricing by 60% by jamroom on Tuesday April 18, 2017 @01:15PM (#14140695)
From the B2 page:

https://www.backblaze.com/b2/cloud-storage.html

"Reliability: 99.999999% durability."

Re:Backblaze B2 Drops Download Pricing by 60% by kcorbitt on Tuesday April 18, 2017 @01:34PM (#14140928)
...but in a single day datacenter, which realistically drops several "9"s from that guarantee when you consider tail events.

I like b2 and am using it for backup in some projects, but unfortunately it's still not an appropriate choice as a primary data store.

The common counterargument I've heard is that you should have multiple redundant storage providers that you manage at the application layer anyway. Problem with that is, there are a near-infinite number of things I "should" do, but very finite resources to do them. S3 backs up across multiple availability zones by default, so using it means one less thing I have to spend brain cycles worrying about.

Re:Backblaze B2 Drops Download Pricing by 60% by jamroom on Tuesday April 18, 2017 @01:52PM (#14141122)
Yep - I agree with you 100%, just pointing out what their page says. I'd love to see B2 get more data centers around the world - when we have to do a 500GB restore on a server in Germany, it's already slow even pulling from S3 - with B2 it would be even slower.
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @02:45PM (#14141644)
Curious - if we open a region in Europe, do people care where?
Re:Backblaze B2 Drops Download Pricing by 60% by killercup on Tuesday April 18, 2017 @02:58PM (#14141798)
I know many German companies require you to store data on servers also located in Germany. (But I have no idea if it's similar in other countries.)
Re:Backblaze B2 Drops Download Pricing by 60% by Symbiote on Tuesday April 18, 2017 @03:34PM (#14142132)
Is it within Germany, or is it just within the EU?

Restricting it to Germany would seem anti-competitive on an EU level, which is usually not allowed. But I don't know the laws.

Re:Backblaze B2 Drops Download Pricing by 60% by rahkiin on Tuesday April 18, 2017 @04:40PM (#14142702)
Depends. But if there are rules about keeping it in-country, you should consider storing it not at a storage provider. Mostly, it is important to keep the data in the EER. So germany is fine. Moving data out of the EU is a big issue, although I have been using B2 but encrypt my data first myself.
Re:Backblaze B2 Drops Download Pricing by 60% by dastbe on Tuesday April 18, 2017 @01:31PM (#14140894)
the OP misquoted S3s durability guarantee. Its 11 9s.
Re:Backblaze B2 Drops Download Pricing by 60% by tonyedgecombe on Tuesday April 18, 2017 @12:33PM (#14140228)
If only you had a Linux client ...
Re:Backblaze B2 Drops Download Pricing by 60% by atYevP on Tuesday April 18, 2017 @12:36PM (#14140267)
Yev from Backblaze here -> For B2 we do have a few Linux connectors! Check them out -> https://www.backblaze.com/b2/integrations.html!
Re:Backblaze B2 Drops Download Pricing by 60% by nerdponx on Tuesday April 18, 2017 @12:36PM (#14140263)
Pretty sure any decent B2 CLI client supports Linux.

But for regular backup, this bit hard. I ended up switching to Crashplan for not only Linux support but also support for external hard drives. Then again the official Crashplan client is GUI-only and AFAIK closed-source, so you can't run it on a headless server

Re:Backblaze B2 Drops Download Pricing by 60% by dublinben on Tuesday April 18, 2017 @12:45PM (#14140379)
As a not incredibly impressed customer of Crashplan, I switched to rclone/duplicati and Amazon Cloud Drive for these and other reasons. The price is the same, and you can use whatever client you want.
Re:Backblaze B2 Drops Download Pricing by 60% by nerdponx on Tuesday April 18, 2017 @12:49PM (#14140408)
Are rclone and duplicati yet more deduplicating backup programs? we now have:

    - rclone
    - duplicati
    - rdiff-backup
    - duplicity
    - attic
    - borg
    - bup
    - zbackup
    - ...?
I'd love to see a comparison of these because right now I have zero idea which one(s) I should use for any given purpose.
Re:Backblaze B2 Drops Download Pricing by 60% by cookiecaper on Tuesday April 18, 2017 @01:41PM (#14141009)
I don't know all of them, but I know some.

* zbackup: Appears to be unmaintained, superseded by attic/borg.

* attic: deduplicating backup repository. Doesn't seem actively maintained anymore either.

* borg: deduplicating backup repository. I used this recently. It's cool, but the performance is disappointing. It's single-threaded. I got into the GitHub a little bit and they seem committed to their current way of doing things, which makes it hard for an external multithreaded compressor to take over (everything is processed in 2 MB chunks).

* rclone: Meant to provide an rsync-like interface to any cloud backend

* duplicity: Automatically performs encrypted, compressed delta-enabled backups, but requires some upfront configuration. Been using this for years.

* rdiff-backup: Haven't used, but it seems more like an ad-hoc backup utility for "I want to keep an old version of this folder". Duplicity seems like a better choice for disk-scale backups.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:41PM (#14141012)
BTW, rclone, duplicati, duplicity, and others now support Backblaze B2: https://www.backblaze.com/b2/integrations.html
Re:Backblaze B2 Drops Download Pricing by 60% by dawnerd on Tuesday April 18, 2017 @01:26PM (#14140831)
Can't speak for the others but I use duplicati and send the backups to my unlimited google drive. I like that it encrypts and gives you the key vs some app saying they encrypt it.
Re:Backblaze B2 Drops Download Pricing by 60% by metafunctor on Tuesday April 18, 2017 @01:32PM (#14140902)
I also ended up switching to CrashPlan. My reasons were Linux support, and the ability to back up to another computer running CrashPlan instead of cloud only. I hate their GUI.

However, we have a sizeable Elasticsearch snapshot in S3, and it's getting expensive. We'd love to use B2 instead, but there's currently no Elasticsearch plugin for B2.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @02:49PM (#14141692)
Running Elasticsearch would require compute, which we don't have. However, how often do you run analysis on the data? Would it make sense to move the data to B2 and put back on s3 when it's needed?
Re:Backblaze B2 Drops Download Pricing by 60% by metafunctor on Tuesday April 18, 2017 @04:14PM (#14142471)
Not sure what you mean that running Elasticsearch requires compute. I'm talking about the snapshot/restore functionality in Elasticsearch. We use it mostly for backups. As far as I can tell, it should be quite possible to implement a B2 plugin for that (Azure, GCS, HDFS, and S3 are available currently).
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Thursday April 20, 2017 @08:17PM (#14162054)
Got it. Let me look into that. Agree that if you're just taking a snapshot from Elasticsearch and putting into B2, that should be an implementalbe plug-in.
Re:Backblaze B2 Drops Download Pricing by 60% by scarface74 on Tuesday April 18, 2017 @01:16PM (#14140719)
Backblaze supports external hard drives. You just have to keep them plugged in. If they aren't plugged in for 30 days, Backblaze will erase the backup - but gives you a warning.
Re:Backblaze B2 Drops Download Pricing by 60% by felixguendling on Tuesday April 18, 2017 @01:31PM (#14140893)
SFTP/SCP/rsync support would be great. Currently, only HTTPS are supported for B2: https://help.backblaze.com/hc/en-us/articles/230383828-Suppo...
Re:Backblaze B2 Drops Download Pricing by 60% by jclimac on Monday April 24, 2017 @05:34PM (#14188533)
@Gleb: I'm from Brazil and if you don't know already bandwidth here is priced like gold, so I just can't ignore the price you are offering.

The service I provide has, currently, an average of 60k http requests daily, approximately 10Gb. Those requests are images stored on Google Cloud Storage. I'm going to offer a new service for photographer to create online galleries, by some calculations I had made the average daily requests will increase around 10x, growing up to 600K daily.

My main questions about B2 are:

1) Is there any use case you know of someone uploading files to B2 directly from the browser? I know this is possible, but as per I know I would have to open tokens and url uploads on JS code, which, obviously, is not cool (and I won't do that). I can do this on S3 and GCS, their API offers the possibility to create a key on the server and use it on the webpage, but I didn't see anything like that on B2.

2) About http requests, is it ok to have this amount of requests (600k) in a daily basis? I mean I know B2 is excellent for backups but I'm not sure if it will be good serving websites that have a big number of images.

Thanks!

Re:Backblaze B2 Drops Download Pricing by 60% by godzillabrennus on Tuesday April 18, 2017 @12:37PM (#14140276)
Glen, thanks for posting!

I'm a backup customer of Backblaze (have been for years) and I backup five of my families computers.

I'd really like it if your online restore feature would let me buy multiple backups on a single hard drive as an option. I've considered buying a hard drive with my backups from your company multiple times but I wouldn't pay the premium you ask without being able to have all five computers on the drive.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:45PM (#14141036)
Thanks for the suggestion (and for being a customer.) We've contemplated adding functionality to send the Backblaze laptop/desktop restore to your Backblaze B2 account. Then you could put multiple restores in a 'snapshot' and get that delivered on a drive. Useful?
Re:Backblaze B2 Drops Download Pricing by 60% by geostyx on Tuesday April 18, 2017 @12:39PM (#14140305)
FWIW, you can return the drive for a full refund.
Re:Backblaze B2 Drops Download Pricing by 60% by infogulch on Tuesday April 18, 2017 @01:21PM (#14140779)
I feel like gp wants to keep a hdd as a third copy of the backup somewhere safe under their physical control.

This is prohibitively expensive to do for multiple computers at once, and gp would prefer a single hdd that has a copy of the backup for multiple computers, which would reduce the cost.

Re:Backblaze B2 Drops Download Pricing by 60% by natch on Tuesday April 18, 2017 @01:56PM (#14141161)
I care less about the download pricing for backups, and more about the storage pricing. I wish you had a comparison for your storage versus Glacier storage, because all of the claims that it is "cheaper than S3" just seem non-credible when I factor Glacier into the picture. It is true that Glacier is a different service from classic S3, but you can't just ignore it. Or at least some of us can't. It does exist. I wonder how it compares. If you don't have a chart, can you tell us here?

I understand download pricing can get complicated. But again I'm not asking about that. I realize download pricing also is important, but right now I'm just asking about storage pricing. Not sure if you are answering questions here but if yes, please share what you know.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @02:28PM (#14141485)
Sure. A bit hard to do as a table here, but here it goes:

                    B2              Glacier
Storage: $0.005/GB/mo === $0.004-$0.005/GB/mo (by region)

Upload: Free === Free + $0.05/1000 transactions

Download: $0.02/GB === $0.05-$0.09/GB

Retrieval Fee: Free === $0.003-$0.03/GB + $0.003-$0.03/1000 requests

Retrieval Time: Milliseconds === 1 min - 12 hours

In short, unless you only upload very large individual files, do nothing with them, and don't access any of them for at least 10 years...B2 is lower cost. (And, of course, doesn't required you to wait to get them out.)

Re:Backblaze B2 Drops Download Pricing by 60% by natch on Tuesday April 18, 2017 @03:07PM (#14141886)
Nice, thanks!

Edit: BTW it's really a breath of fresh air to see someone whose product is covered hanging around for a while answering questions. So many people in your position either don't show up on their HN stories at all, or do so only for an hour or so and then leave a ton of questions unanswered.

Probably my number one wish list feature for hacker news would be if they could somehow coax people at companies that get linked to return more, including beyond the first hour or first day.

I hope you check back on this story later and answer more questions that come in from different time zones.

Re:Backblaze B2 Drops Download Pricing by 60% by atYevP on Tuesday April 18, 2017 @07:20PM (#14143961)
Yev from Backblaze Here -> My #1 feature request would be notifications or something a-la reddit that would tell me if someone replied to me. Scrolling through these is tough!
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @06:57PM (#14143787)
Hard to keep up with all conversations on-line, but HN tends to be high signal-to-noise, and I appreciate the feedback we get.
Re:Backblaze B2 Drops Download Pricing by 60% by unixhero on Tuesday April 18, 2017 @12:31PM (#14140211)
Great news!

Maybe now I can consider it for my 10tb offline backups.

Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @01:45PM (#14141048)
What software do you use to actually do the backups/manage them? (Wondering if we'd need to work with someone else to integrate to make that easier for you.)
Re:Backblaze B2 Drops Download Pricing by 60% by unixhero on Friday April 21, 2017 @02:05AM (#14163421)
Borg
Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @02:21PM (#14141411)
Another IMPORTANT aside. Very annoying. Backblaze requires an authorize() call to obtain a token before calling any API function. Please see https://www.backblaze.com/b2/docs/b2_authorize_account.html.

This is unlike S3, which simply needs a STATIC API key, and boom, you can access the API. I don't know why you did this, but it is bad developer UX. Is it because tokens are "cool". I see no need to have this concept of tokens for your API. Why I am annoyed:

1) An API operation now requires one ADDITIONAL call. B2 already requires 2 calls to upload a file, and this now means 3 calls!

2) Related to above, this means bad latency. The roundtrip can be significant, especially since you only have 1 data center in California. My users will NOT be happy.

3) My code is much more complex now.

4) It is more work FOR YOU. This means more processing. More resources wasted FOR YOU. More cost. Why would you do this? You are supposed to be lean. Imagine millions of extra calls EVERY day. Adds up.

Is there a solution to this? Can the token be cached? If so, for how long.

Also, would love to see a reasonable rationale for this? If there is no rationale, then please provide a STATIC approach to access your API.

I REALLY like B2, and I wish they could think of developer UX a little more.

Re:Backblaze B2 Drops Download Pricing by 60% by jjeaff on Tuesday April 18, 2017 @02:54PM (#14141749)
Somewhere on their site, they address this issue. It is part of the reason they are able to offer cheaper prices. By having to request the token first, then upload, this allows them to avoid using load balancers. The less equipment they have to have, the cheaper your storage can be.
Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @03:07PM (#14141883)
I do not believe this is for the authorize() token issue. Rather, this is relating to the fact that an upload requires to calls:

https://help.backblaze.com/hc/en-us/articles/218513487-Is-th...

Re:Backblaze B2 Drops Download Pricing by 60% by skj on Tuesday April 18, 2017 @03:29PM (#14142072)
No, N uploads require N+1 calls.
Re:Backblaze B2 Drops Download Pricing by 60% by loxias on Tuesday April 18, 2017 @02:53PM (#14141737)
Uh, what's your complaint? Why in the world would you assume that it has to have the same API as S3? I also don't understand your comments about "cool". You need an auth token so that the subsequent API calls, well, know you're authentic... Makes perfect sense to me.

Yes, the first API call requires authenticating first but every additional call reuses cached credentials, so no change to latency.

I've been using Backblaze to write a for-fun app, using c++ and libcurl, for the last few weeks and I've found it to be wonderful from a developer perspective.

Their documentation is short and sweet. They have capabilities to help you stress test your implementation to make sure you handle edge cases! (you can pass an extra flag and it will randomly send errors) That is so amazingly helpful! And to top it off, their support staff is quick and responsive even with technical details. _And I don't even have a paid account_

If you find dealing with this to make your code "much more complex", then as they say "my consulting rates are very reasonable, my email is in my profile". :P

(Disclaimer: I do not work for Backblaze, have never given them a dime of my money. So far.)

Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @03:00PM (#14141815)
I listed very valid concerns regarding why I was upset. I certainly know what auth tokens are.

I like there docs, short and sweet. It fits in their lean approach. But, using tokens is much heavier than the easier static key approach.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @03:46PM (#14142230)
Brian from Backblaze here.

> Backblaze requires an authorize() call to obtain a token before calling any API function.... this now means 3 calls to upload a file

That is correct if you only plan to upload one file. The "authorize" can be reused for weeks, and so can the upload URL. So if you want to upload 100,000 files, it is 100,002 calls to upload the 100,000 files (give or take).

It is true this is slightly more complex than it could be, but in turn this means we don't need to have any load balancers in our datacenter (which in turn makes the cost of the service cheaper and at the same time enormously more easily scalable). The idea here is that your application authenticates, then you ask for a URL to a vault that has spare space on it. Then your application "hangs up" (stops talking with that server) and contacts the vault directly. This means there is zero "chatter" between our vaults and that you are talking directly (no hops) to the vault you are storing your data on.

Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @03:59PM (#14142332)
Thanks Brian. I don't see the mention of authorize "can be used for weeks" in the docs. I see "at most 24 hours". Please see https://www.backblaze.com/b2/docs/b2_authorize_account.html
Re:Backblaze B2 Drops Download Pricing by 60% by d4l3k on Tuesday April 18, 2017 @02:36PM (#14141561)
If you look at their API you'll see that the token can be cached for up to a week. I doubt that having a check if the token is up to date will add much complexity.
Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @02:44PM (#14141629)
It really does add complexity:

-I have to store the cached token somewhere:

-I have to do date arithmetic, what if my server clock is incorrect,

-if an API call failed, I have to check for an additional error: a invalid token

I don't mind complexity when it is needed, but I do not prefer complexity for no good reason.

That being said, can it really be cached up to a week? I see "at most 24 hours", which means it could be even less than 24 hours, so there's no guarantee. Please see: https://www.backblaze.com/b2/docs/b2_authorize_account.html

Re:Backblaze B2 Drops Download Pricing by 60% by FireBeyond on Tuesday April 18, 2017 @03:05PM (#14141860)
> -I have to do date arithmetic, what if my server clock is incorrect,

How is your clock inaccuracy in any way the responsibility of a service provider (Backblaze or other)? From certificate expiry to inaccurate logging, and so on, complaining about "what if my clock is wrong, I have to date math in my app!" is precisely "solving the wrong problem".

Re:Backblaze B2 Drops Download Pricing by 60% by loxias on Tuesday April 18, 2017 @02:58PM (#14141796)
"-if an API call failed, I have to check for an additional error: a invalid token"

In general it's considered good practice to always check return values and error codes. Not for all code, but just for the code you would like to work.

Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @03:09PM (#14141909)
@FireBeyond Did you even read my reasoning? All I'm saying is that there is ADDITIONAL complexity know in relation to the other, simpler static key approach.

Not shifting blame on Backblaze for my server clock being potentially inaccurate.

Re:Backblaze B2 Drops Download Pricing by 60% by truetraveller on Tuesday April 18, 2017 @03:03PM (#14141836)
@loxias. I do not understand your response. Of course one should check error codes. My point is: now there's one more thing that can go wrong and needs to be checked.
Re:Backblaze B2 Drops Download Pricing by 60% by sfaruque on Tuesday April 18, 2017 @01:42PM (#14141019)
Hi Gleb,

I'm about to sign up for the B2 Cloud Storage, just noticed a tiny typo in the "Keep costs down" section on https://www.backblaze.com/b2/cloud-storage.html:

"...the fist 1 GB of downloads per day are free."

Re:Backblaze B2 Drops Download Pricing by 60% by aschampion on Tuesday April 18, 2017 @01:57PM (#14141183)
Just read it in a kiwi accent and all is fine.
Re:Backblaze B2 Drops Download Pricing by 60% by atYevP on Tuesday April 18, 2017 @07:23PM (#14143989)
Yev from Backblaze here -> We got it fist! ;-)
Re:Backblaze B2 Drops Download Pricing by 60% by budmang on Tuesday April 18, 2017 @02:50PM (#14141709)
Thanks. Will get that fixed today.
Re:Backblaze B2 Drops Download Pricing by 60% by finnh on Tuesday April 18, 2017 @02:18PM (#14141380)
Not quite what you are asking, but: what colo facilities have max bandwidth to you, and what is my likely MB/s at the top-performing location?

S3 can deliver 500 MB/s to a single EC2 node pretty consistently (same region, multiple streams) which keeps me there.

Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @03:58PM (#14142326)
Brian from Backblaze here.

> what colo facilities have max bandwidth to you

Our primary colo is SunGard in Rancho Cordova, California. I believe we are currently provisioned for about 220 Gbits/sec (in this one facility alone). I'm not completely sure what this would top out at if we asked for more, but it is a colocation facility with a bunch of network providers such as Cogent, GTT, Century Link, etc. I kind of expect they can provide anything we ask for?

> what is my likely MB/s at the top-performing location

I have three answers, you can choose one. :-)

1) If you want to download a file once after a very long time of not accessing it (meaning it is not in the "B2 cache"), and with one thread, it must be fetched from the vaults to the "caching layer". Spooling a single file off of the vaults with one thread is relatively slow at about 15 Mbits/sec.

2) If the file is already in the caching layer, then it is fetched from local SSDs already inside the caching servers. Each individual caching server has a 10 Gbit/sec network card so if you only have one thread then you are absolutely limited to 10 Gbit/sec (and realistically the caching servers are not completely idle so a realistic number might be only 3 or 4 Gbit/sec).

3) If you want to fetch many files with many threads, fetch one file at a time with each thread. I'm not sure how many HTTPS threads an EC2 node can handle, maybe 200 safely? If it isn't in the cache then you get 200 * 15 Mbits/sec = 3,000 Mbits/sec = 375 MBytes/sec. If it is coming out the cache even faster?

Hope that helps!

Re:Backblaze B2 Drops Download Pricing by 60% by finnh on Tuesday April 25, 2017 @11:56PM (#14200324)
It does - thanks!
Re:Backblaze B2 Drops Download Pricing by 60% by monort on Tuesday April 18, 2017 @04:13PM (#14142463)
I'd like to check checksums for my data, but it's still too expensive to do regularly. Can you provide an api for doing it on your side?
Re:Backblaze B2 Drops Download Pricing by 60% by brianwski on Tuesday April 18, 2017 @06:26PM (#14143582)
Brian from Backblaze here.

If you use b2_list_file_versions (or any of the b2_list functions) there are checksums included in that list (SHA1).

When you ask for that file list, the results are actually coming out of a large Cassandra cluster (database with many computers and fast SSDs).

Ok, so even if you do not request a file list of any kind, Backblaze itself automatically is always walking over the entire data farm checking every single last file's SHA1, plus checking if the files match the database's SHA1. A full sweep completes in less than a month then begins again.

If any bit error is detected (which happens semi-regularly since we have over 72,000 hard drives and bits get flipped) then we rebuild the files from the other copies (Reed-Solomon encoded across many separate computers in many separate locations in our datacenter).

We love and take care of your data so you don't have to.

Re:Backblaze B2 Drops Download Pricing by 60% by monort on Wednesday April 19, 2017 @03:18PM (#14150856)
Thanks for the information! Though I'd like to be able to initiate the sweeping process for my files more often, maybe you can charge for such requests?