News for hackers. Stuff that matters.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ksec on Monday May 04, 2020 @12:34PM (#23070092)
Backblaze is also the founding member of Bandwidth Alliance, meaning getting those B2 via Cloudflare is essentially free.

So you are only paying for Storage. ( Correct me if I am on this one )

I wonder why doesn't ALL non HyperScale Cloud Vendors, like Linode and DO provide one click third party backup to B2. You should always store offsite backup somewhere. And B2 is perfect.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @01:00PM (#23070474)
Yev from Backblaze here -> That's correct. We're a founding member of the Bandwidth Alliance and you can find more info about that on our Blog (https://www.backblaze.com/blog/backblaze-and-cloudflare-part...) or FAQ (https://help.backblaze.com/hc/en-us/articles/217666928-Using...). May the 4th be with you ;-)
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by drenvuk on Monday May 04, 2020 @01:33PM (#23070902)
Hey I've been wondering this but are the transaction costs also waived or is it just the bandwidth fees? I feel like if both are gone it'd be hard to make a profit...
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @01:54PM (#23071138)
For that integration it's just the download fees. The transactions do still get billed, but many people can stay within the free daily allotment.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by q92z8oeif on Tuesday May 05, 2020 @09:30AM (#23079811)
What's the point of the Alliance in this case?

if I use against-the-alliance corp, i will just pay the storage+download from backblaze, just the same as using in-bed-with-alliance corp.

Or does the alliance will ensure i only get charged for 1download per file per billing cycle, no matter if the CDN cache can't hold all the data and they download it many times? Does the alliance ensure (and more importantly, who monitors) it?

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Tuesday May 05, 2020 @10:31AM (#23080383)
Hey good question, it's essentially a peered connection so you're not paying for the transfer between providers, you can learn more here -> https://www.backblaze.com/blog/backblaze-and-cloudflare-part...
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by TechBro8615 on Monday May 04, 2020 @02:10PM (#23071326)
Where can I find the pricing info for this allotment?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @02:51PM (#23071816)
All of our pricing is available on the site (no need to call anyone) - B2 Cloud Storage is listed here -> https://www.backblaze.com/b2/cloud-storage-pricing.html. Just scroll down for the transactions!
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by dxxvi on Monday May 04, 2020 @05:47PM (#23073706)

                                    Storage        Download
    BackBlaze                       .5 cents/GB    1 cent/GB
    S3 (one zone infrequent access) 1 cent/GB      9 cents/GB
I'll transfer my 250GB of videos and images from S3 to BackBlaze some day.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @05:53PM (#23073775)
Heh, we make it easy! Check out this Flexify partnership -> https://www.backblaze.com/blog/supercharged-data-migration-w...
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by rossjudson on Monday May 04, 2020 @09:32PM (#23075431)
What does "Charged for any portion of a GB." mean?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by KMnO4 on Tuesday May 05, 2020 @08:51AM (#23079452)
It means 1.01gb, 1.1gb, and 2.0gb are all charged the same
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by rossjudson on Tuesday May 05, 2020 @01:23PM (#23082428)
What about 0.1gb, 0.5gb? I guess that's my question. Is this a minimum charge per download?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Wednesday May 06, 2020 @03:57PM (#23095572)
Yev here -> No there's no minimum charges for downloads, it's just $0.01/GB down and $0.005/GB/Mo for storage, but everything is calculated on a byte hour level :)
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by rossjudson on Friday May 08, 2020 @03:25PM (#23118458)
Thanks! That's where I was curious.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anotheracct_ on Monday May 04, 2020 @03:40PM (#23072395)
Isn't this incompatible with Net Neutrality?

I think Cloudflare already was against net neutrality, but people who believe in that principle might need to avoid Backblaze as well if that is the case.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by manigandham on Monday May 04, 2020 @03:57PM (#23072570)
No, this has nothing to do with net neutrality. Your ISP isn't discriminating what data you consume, whether it's from cloudflare or another CDN.

This is about the fact that bandwidth is usually bought by capacity between networks rather than usage and they're passing those savings on to you instead of charging expensive per-GB costs like the major clouds.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anotheracct_ on Monday May 04, 2020 @04:00PM (#23072601)
I'm referring to Cloudflare as the network provider. It seems not-net-neutral to be charged different amounts depending on the relationship between the storage provider and the network provider.

If the argument is a semantic one, that net neutrality is so narrowly defined that it doesn't apply to them as a CDN rather than a conventional ISP, it still stands that the situation is not-net-neutral if we generalize that to include CDNs, which it ought to, since that's what the N stands for.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by toast0 on Monday May 04, 2020 @04:45PM (#23073033)
Network neutrality is a difficult concept to apply because its core concept doesn't map well to reality.

The core concept, in my mind, is that all packets should be treated equally, and should cost the customer the same amount per byte.

The problem is that different packets take different network paths and have different costs to your provider. The bandwidth alliance partners are exchanging traffic via peering links, so there's only the fixed costs of equipment and any port charges/interconnection fees to the peering facility. Packets sent to other destinations may pass through a paid transit link and cost the provider. If your provider charges you the same amount for both types of packets, they're not being transparent about their network costs which is bad for you, and is also bad for them, because you may be able to adjust your traffic so more is settlement free and less is paid --- but you won't do that without an incentive.

Of course, if you don't like how Backblaze manages its network, there's a bunch of other network storage vendors who you could switch to (many of which are members of the Bandwidth Alliance). Or, it's not too hard to build a storage box or two and get them into a colocation somewhere.

For your residential ISP though, for most people, if you don't like the network policies, you might have another option, but they will likely have similar policies. You don't have meaningful choice in a single residence, and moving residences to get better choices isn't a meaningful option either. So, network neutrality is a policy to (attempt to) regulate residential ISP behavior, in order to provide a reasonable policy for users. It's still a problem that it doesn't match reality, and it doesn't provide user choice, but it has gotten a lot of support.

I would rather see mandatory line sharing for residential ISPs, it's a lot easier to define and with proper regulation offers a path towards consumer choice, and gives people a way to take action if their provider has poor network policies --- if your provider runs an acceptable last mile service, but provides poor interconnection to the rest of the world, you could become a line sharing partner and provide better interconnection to the world without having to build out an overlay last-mile network (which is incredibly capital intensive and difficult)

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by fragmede on Tuesday May 05, 2020 @06:05AM (#23078316)
Is it really that difficult a concept? Comcast charging Netflix for access to Netflix's customers, when Netflix's customers are paying Comcast for Internet access, is bullshit.

Reword it to be generic, and dress it up in more polite language, but that's what it is.

Comcast, nor any other ISP, doesn't get to shake down Netflix or Google or Facebook nor any service providers, big or small, on the Internet.

Sure, behind the scenes there are backbone providers and peering arrangements and BGP, but those are technical details. Network neutrality means as a consumer I pay an ISP and get connected to the Internet.

Whereas Comcast is a cable TV company, the game between companies of consumer chicken is not theoretical, eg

https://www.thedenverchannel.com/news/local-news/why-you-sti...

https://mynbc15.com/news/local/att-and-directv-have-removed-...

https://www.washingtonpost.com/business/cbs-blackout-on-dire...

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by toast0 on Tuesday May 05, 2020 @02:00PM (#23082819)
> Is it really that difficult a concept? Comcast charging Netflix for access to Netflix's customers, when Netflix's customers are paying Comcast for Internet access, is bullshit.

I agree, that it's bullshit, but please consider:

A telephone company charges me to have a phone line and per minute to place and receive calls, and charges whoemever I'm conversing with to have a phone line, and to call me.

If Comcast wasn't large enough for Netflix to want peering, Netflix would probably use paid transit to get there (they might even use paid transit that Comcast would also have to pay for).

So, the reason it's bullshit can't be because Netflix has to pay for a service that Comcast is providing; because paying for network services is the norm. The reason also can't be because Comcast is charging both ends of the connection for the service it is providing to both, namely connecting the two parties; because both sides paying for connectivity is the norm in telecommunications.

There's some other reason it's bullshit. A big factor, of course, is that internet video services compete with the ISPs video services, and there's some implicit unfairness there. I would say it's because the residential customer is stuck with a very limited set of choices, and can't pick an ISP that distinguishes itself with open peering. It might be because the norm for internet peering is if there is a significant imbalance in traffic, the party sending the most pays; so the residential ISPs should be paying their residential customers; this one doesn't quite work, because it's not actually a peering relationship, and the norm is transit customers pay the transit provider for traffic in either direction.

> Sure, behind the scenes there are backbone providers and peering arrangements and BGP, but those are technical details. Network neutrality means as a consumer I pay an ISP and get connected to the Internet.

The problem here is there is no "the Internet" to connect to. An ISP needs to connect to substantially all of the other networks; and while a small ISP may simply use a single upstream ISP, where it would be easy for the small ISP to be neutral (bits in = bits out), a larger ISP is going to have a diversity of connections, and that's going to lead to defacto non-neutrality as some connections are bigger than others, and some connections are longer than others, and some connections are more expensive than others.

I fully understand the desire to restrain residential ISPs (and mobile ISPs) from anti-competitive behavior; I guess network neutrality might work for that.

But, when you apply it to other networks, it doesn't seem to make sense. In this case, you have a group of networks that would like to lower costs, both for themselves, and their customers, and they've agreed to settlement free peering, and to also not charge customers for bandwidth on the settlement free links. Network neutrality doesn't really speak towards the interconnection, but says that customers should be charged the same rate for all bandwidth, even when the underlying costs are different. I don't understand why that's a good thing in this case?

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by fragmede on Tuesday May 05, 2020 @06:18AM (#23078384)
Mandatory line sharing is an interesting idea. We've tried it before, back in the DSL era (2000s). Not saying we shouldn't try it again, but it only sorta worked. Whether instructions came down from ATT corporate offices or were malice, or incompetence, on the part of locally contracted installing technicians; the shared boxes were the worst of the commons.

If a neighbor got from a rival DSL provider, your Internet was liable to go out until your ISP was able to get a truck out to fix the damage to your connection that the competition's technicians did.

Those laws are still around, but unfortunately, (A)DSL tops out at single-digit megabits. An upgrade from dial-up, but not competitive with todays broadband market for those with other options.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by toast0 on Tuesday May 05, 2020 @02:14PM (#23082982)
I'm pretty aware of the issues with past implementations of line sharing; a competent regulator would be required to iron out these issues. Of course, network neutrality requires a competent regulator as well.

Pricing was a big issue in the past --- where the 'wholesale' per-line tariff charged to the competitive carrier was more than the retail price charged to incumbent customers.

I think, in most cases, new install breaks other user types of issues are related to poor records of which line is used by which customer; that happens within the incumbent as well though (when I got ADSL2 installed, one of my neighbor's connections went out, and pretty soon there were three AT&T trucks on the street to work everything out). That particular issue could probably be solved by allowing the incumbent to manage all the installs and monitoring for install-time customer steering.

The federal mandate no longer covers line sharing, it only covers line leasing for copper telephone service; wherever your premises is directly connected to, if there's space for competitive equipment, the incumbent must make it available. Of course, most telephone companies have moved customers to remote terminals for better speeds, and there's no room for competitive equipment in the remote terminals; and a lot of telephone companies are replacing copper networks with optical networks, and those aren't covered either. The whole concept got submarined by the insistence that it apply to telephone networks and not cable or other "new technology" networks, and then deciding not to apply it anywhere in light of court cases that applying to one and not the other was unfair.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by manigandham on Monday May 04, 2020 @05:22PM (#23073440)
Cloudflare doesn't charge for bandwidth, it's as neutral as it gets.

The charges are from the cloud (storage) providers. And most of them charge for bandwidth usage, but they don't discriminate based on what data you send. It doesn't matter if it's source code, photos, or linux binaries so there's no net-neutrality issue here.

You can potentially argue that Backblaze is discounting traffic based on the upstream network but that's not active discrimination. There are hard costs to internet transit and some companies can offer better pricing depending on where the traffic goes. For example, downloading from Australia is more expensive than downloading in America because of the infrastructure costs, and this isn't considered a net-neutrality violation.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by khc on Monday May 04, 2020 @04:24PM (#23072830)
cloudflare is not charging different amount for different storage providers, some storage providers are waiving the fee when the upstream is cloudflare.

disclaimer: ex-cloudflare

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anotheracct_ on Monday May 04, 2020 @04:43PM (#23073019)
So, Backblaze is not-net-neutral.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @05:44PM (#23073687)
As far as our service is concerned everyone pays the same price for B2 Cloud Storage, $0.005/GB. We're not a network provider and in this case neither is Cloudflare. We are both service providers. The partnership between us simply allows people to move data between us freely, but you still need to pay Backblaze B2 for the storage of the data, and you need to play Cloudflare for the distribution!
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by NikolaeVarius on Monday May 04, 2020 @06:05PM (#23073897)
You are ignoring the responses to your original point. This is not relevant to net neutrality.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anotheracct_ on Monday May 04, 2020 @07:28PM (#23074600)
I think I addressed that comment, actually!
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @04:23PM (#23072824)
Yev here -> unsure how those are related, but you certainly don't NEED to use Cloudflare if you're using B2 Cloud Storage as the data origination point, it's just an option that was brought up in the thread!
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anotheracct_ on Monday May 04, 2020 @04:46PM (#23073051)
It sounded like the price would be different if you used a certain network provider, which is not-net-neutral.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by atYevP on Monday May 04, 2020 @04:53PM (#23073120)
I believe in this case Cloudflare and Backblaze would be considered service providers, not network providers (like Comcast or AT&T, etc...). And you'd still need to pay each service individually for the service they're providing. It's an alternative to say, Amazon S3 and their own CDN which also does not charge for egress in between the two services (since they own both). But in our case, you pay Backblaze B2 Cloud Storage of the storage, and you pay Cloudflare for their CDN capabilities, the partnership simply makes the transfer between our storage and their CDN free.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by sp332 on Monday May 04, 2020 @04:55PM (#23073138)
Transit does cost different amounts of money depending on who you negotiate transit with. That's not part of net neutrality.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by brainless on Tuesday May 05, 2020 @02:18AM (#23077067)
Airlines have alliances, it does not stop you from taking any airline you want for your route, but costs may vary.

It is just a property of networks. As an end user you still get to use all providers, but two of them may use a dedicated connection to reduce internal prices (the price benefit of which they may or may not give back to the user).

Neutrality from the point of the user is not affected when it comes to service being available.

Now if one airline alliance said you are not allowed to get on their flight if you hoped off from a rival alliance, then we have an issue.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by hangonhn on Monday May 04, 2020 @02:50PM (#23071804)
This is so mind-blowing-ly crazy and awesome. I've always loved the idea of Backblaze but never tried it. But this combination is so insanely good and valuable. When an efficient market/economy works correctly, the results are just amazing: Cloudflare and Backblaze essentially got their unit costs for their respective products so low that for just a small money fee, I can do something that would have been prohibitively expensive maybe just a decade ago.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ksec on Monday May 04, 2020 @03:29PM (#23072269)
On the note of unit cost I doubt it will be much lower in the near future. Cost / GB isn't falling much if at all, and their Load balancer are already eating into their margin which means any further reduction in unit cost are only gaining back what they took in with the S3 API. Hopefully the new S3 compatible API means they could make all that back with volume.

But this is also a sad realisation that storage tech aren't getting cost reduction any further.

( Of course Blackblaze can always prove me wrong :D )

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by toohotatopic on Monday May 04, 2020 @03:56PM (#23072556)
Wasn't there a startup on hn some weeks ago that used a crypto mining approach to server management that was much cheaper?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ADefenestrator on Tuesday May 05, 2020 @01:21AM (#23076783)
Disclaimer - Backblaze employee here, but just speaking for myself:

It was.. sort of cheaper. They didn't actually build the servers, and as described the server wouldn't work (onboard SATA didn't support port multipliers, lack of ECC would probably cause problems in practice, bit hand-wavey on power/space/network/manpower costs, etc). The goal of the article was to get other people to build cheap storage and put it up for rent on their network. They do have some amount of storage space available for very cheap on the network now, but personally I suspect it's people who figured "what the heck, I'll give it a try!" as opposed to people actually building storage servers and making a profit renting them out.

I was honestly pretty disappointed - I'd hoped they'd found a cheap motherboard with ECC and support for port multipliers, but nope.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Dylan16807 on Tuesday May 05, 2020 @02:49AM (#23077223)
With the client responsible for chunking, redundancy, and error correction, I don't think the lack of ECC really matters for their use case. The rest of the issues are more important.

Edit: The motherboard they picked does support ECC memory, anyway. In general ASRock models do.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ADefenestrator on Tuesday May 05, 2020 @12:20PM (#23081690)
I haven't dug too deeply, but it's unclear whether anything but the Asrock Rack boards have full validated ECC implementations for Ryzen (vs just using the memory but ignoring ECC), and I think that may depend some on which Ryzen CPU is used (maybe PRO-only). I'd love a source of better info there, though.

You're right though - since the client's doing the work and they have a lot of redundancy/diversity in the storage it's not as big of a deal for them as it would be for us. I'd be a bit wary because the client-only verification does mean that there's no verification-with-ECC step in the entire chain, but I'm not sure that's significantly worse in terms of actual risk.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by justinclift on Tuesday May 05, 2020 @05:30AM (#23078120)
How far off do you reckon Backblaze is from designing your own motherboards + electronics, with just the pieces you need? :)
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ADefenestrator on Tuesday May 05, 2020 @12:05PM (#23081529)
Pretty far. Since we cram so many drives into each server, our total server count is actually relatively low for the amount of storage we have. I'm not sure exactly how many units you need to amortize the design costs across to make it worth it for a custom ODM design, but I suspect it's in the tens of thousands.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ksec on Monday May 04, 2020 @05:30PM (#23073529)
Yes and it had little redundancy with limited up time and speed guarantee, with no S3 API compatibility. And if I remember correctly even that was $0.003 per GB, with lots of unit cost that a lot of people including me thought to be miscalculated, where it should be closer to $0.4 for it to be somewhat break even.

So may be it is cheaper, but not really comparable.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Hamuko on Monday May 04, 2020 @12:38PM (#23070150)
How does that work? Can I just dumb two terabytes of video into Backblaze B2, setup a Cloudflare account and have people watch those videos with it costing me only $10 a month? Because that doesn't sound right.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by heipei on Monday May 04, 2020 @01:00PM (#23070475)
The Cloudflare ToS explicitly exclude that use-case.

2.8 Limitation on Serving Non-HTML Content The Service is offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as a part of a Paid Service purchased by you, you agree to use the Service solely for the purpose of serving web pages as viewed through a web browser or other functionally equivalent applications and rendering Hypertext Markup Language (HTML) or other functional equivalents. Use of the Service for serving video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by mikehearn on Monday May 04, 2020 @01:28PM (#23070835)
In a previous HN post [1] where someone wrote up how to use B2 as an image host, the Cloudflare CEO chimed in and addressed rule 2.8 specifically, saying if Cloudflare workers are used for URL prettifying and redirecting, a different ToS is applied and that use-case would be fine.

Does that mean your video use-case would also be fine? I have no idea. An HN comment from the CEO doesn't seem like it would hold up if Cloudflare suddenly shut down your free account.

I'd love for Cloudfront to officially clarify the limits of the Cloudfront/B2 alliance in terms of external traffic. The confounding issue here is that B2, as a storage service, is not really intended for "serving web pages and websites" — it's for larger files, binaries, etc. — and therefore any traffic from B2 going through Cloudfront is sort of de facto in violation of 2.8.

[1]: https://news.ycombinator.com/item?id=20790857

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by brianwski on Monday May 04, 2020 @02:05PM (#23071273)
Disclaimer: I work at Backblaze so I'm biased. :-)

> B2, as a storage service, is not really intended for "serving web pages and websites" — it's for larger files, binaries, etc

It might be missing a couple features (which is a pet peeve of mine) but we SURELY intend for it to be used for serving web pages. That's one of the largest differences between "Backblaze Personal Backup" (our original product line) and Backblaze B2. The largest parts of the redesign/refit when we originally did B2 was around the concept of what we call "Friendly URLs" (web page names, folder names) instead of just ugly 82 character hexadecimal file names like Backblaze Personal Backup stores all your files in.

For full disclosure, Backblaze B2 isn't a great "hosting" solution for something like WordPress because we lack two or three things, one of which is comically easy to fix and I keep trying to convince everyone to do it. The issue is URLs that end in a "/" (trailing slash) basically need to "guess" that after that is an ".html" or ".php" or whatever. So the URL: https://f001.backblazeb2.com/file/ski-epic-c/full/2015_scotl... does not work, but the URL: https://f001.backblazeb2.com/file/ski-epic-c/full/2015_scotl... does work. All modern web servers do this automatically filling in of the "index.html", but it is missing from Backblaze B2 currently. And it would take just a day or two for one of our developers to fix it. And dang it, I'm going to get it done one of these days.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by john-shaffer on Monday May 04, 2020 @03:29PM (#23072267)
S3's use of separate servers for website hosting is actually very sensible. Options like usage of index.html and error.html only apply on the website servers and won't cause any surprises for people using the service as a key-object store.

That said, I would absolutely not consider using B2 without support for index.html, error.html, and Website-Redirect-Location.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by erichocean on Monday May 04, 2020 @09:34PM (#23075439)
You can trivially use Cloudflare Workers to implement that functionality on top of B2.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by mahesh_rm on Monday May 04, 2020 @03:06PM (#23071984)
Please be inspired by how Netlify handles url rewriting through their netlify.toml file, in doing so, if it will be possible. :-)
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Dylan16807 on Tuesday May 05, 2020 @02:38AM (#23077170)
Sounds like workers are fine in general, especially on the non-free tier that only increases the budget by $5 a month.

That covers 10 million requests, which could each be 10MB chunks, or 512MB cached files, or possibly larger.

(A raw mp4 needs about 3 requests to start plus one per seek.)

But I wouldn't be surprised if that doesn't scale to enormous amounts of data.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by judge2020 on Monday May 04, 2020 @05:08PM (#23073276)
To clarify, the key line is "a disproportionate amount of non-HTML content". Serving videos isn't prohibited but CF still has to prevent free plan websites from becoming huge loss-leaders.

Another big asterisk is that this applies to all proxied (DDOS-protected) content, not just the ones that use the CF cache. CF pays for all of their uplink bandwidth out-of-pocket regardless of if it's cached. You can see what happens when you proxy multiple terabytes of content on the free plan in this thread[0] (again all proxied bandwidth costs CF, which is why this user had their zone unproxied).

0: https://community.cloudflare.com/t/the-way-you-handle-bandwi...

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by DoctorOW on Monday May 04, 2020 @01:09PM (#23070592)
From what I can tell that's more about their caching service. It mentions exceptions for hosting from a paid service and correct me if I'm wrong but wouldn't Backblaze be that paid service?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by jjeaff on Monday May 04, 2020 @01:21PM (#23070736)
They are talking about their own paid service, obviously. I don't think they care whether you are paying for someone else's product or not.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by celicaraptor on Monday May 04, 2020 @01:22PM (#23070752)
Perhaps they are talking about a paid service that they offer,like Cloudflare Stream https://www.cloudflare.com/products/cloudflare-stream/
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anon102010 on Monday May 04, 2020 @01:23PM (#23070768)
This is huge if true - folks keep on saying Cloudflare can replace cloudfront for free. If we could put 20TB of binary content for software distro / videos etc on B2, then stream by cloudflare I'd actually believe the claims.

Every time we've tried this with other "free" providers there always seems to be "fine print" when you start pushing huge bandwidth on these "unlimited" plans. It seriously is not worth the time at some point.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Dylan16807 on Monday May 04, 2020 @01:02PM (#23070514)
Cloudflare's main service excludes video.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by tpmx on Monday May 04, 2020 @02:24PM (#23071475)
This is key for many.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by chx on Monday May 04, 2020 @01:24PM (#23070784)
The way I understand, you can do this with images, eastdakota confirmed that here earlier https://news.ycombinator.com/item?id=20791660 not sure about videos.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by celicaraptor on Monday May 04, 2020 @01:23PM (#23070767)
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by varikin on Monday May 04, 2020 @12:42PM (#23070226)
I believe you still pay Cloudflare costs, but the traffic between Cloudflare and B2 is free on both platforms. But it might be worth double-checking the fine print.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Hamuko on Monday May 04, 2020 @12:44PM (#23070254)
Isn't the Cloudflare CDN included in the free plan as well?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by georgyo on Monday May 04, 2020 @12:56PM (#23070420)
It is, but they cap out the file size they will cache on the free plan to be 512MB. So you would need to chunk up your videos to get free bandwidth.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by derefr on Monday May 04, 2020 @02:13PM (#23071373)
> So you would need to chunk up your videos to get free bandwidth.

Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no?

Is there some use-case where people are currently serving 400MB contiguous video files from a CDN? I can't think of one. YouTube doesn't. Netflix doesn't. Even porn sites don't.

I guess Archive.org has some large video files on there in various places, that can be direct-downloaded; but the recommendation Archive.org itself makes, is to consume those via BitTorrent. Presumably they don't have a CDN partner willing to handle their unique workload for cheap.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by jcampbell1 on Monday May 04, 2020 @03:24PM (#23072220)
You don’t need DASH or HLS to seek to an arbitrary point quickly. The mp4 container format has an index which stores playhead time -> byte. This index is either at the end of the file or beginning, and tools like qt-faststart move the index to the beginning, which makes videos start much quicker when serving over http. Browsers will use the index to issue range get requests an be able to seek just fine.

Serving a 400mb video via a CDN is highly dependent on the CDN. Some will construct a cache entry from a slop of range get requests and translate them to get the missing pieces and work brilliantly and other CDNs should be avoided.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by chx on Monday May 04, 2020 @06:38PM (#23074209)
ffmpeg.exe -i input.mp4 -c copy -movflags faststart -y output.mp4
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Dylan16807 on Monday May 04, 2020 @02:57PM (#23071876)
> Having a functional video player on your site or in your app (e.g. one where you can skip to arbitrary times without requiring the video be buffered up to that point; or where a video can be "resumed" from the middle if you leave it and come back) already requires that you use MPEG-DASH or HLS; which in turn implies/necessitates pre-chunking, no?

Browsers are smart. They only buffer a few megabytes at a time and can seek around pretty efficiently.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by derefr on Monday May 04, 2020 @03:06PM (#23071978)
That requires the video to be encoded in a way where you can just start reading the stream from any random byte offset, and everything will still work. Video files are not usually encoded this way (any more.) Resume an MP4 or MKV video half-way through, without reading the TOC-ish stuff from the first chunk, and you'll get garbage that maybe resyncs after 20 seconds.

It's totally possible to "encode for streaming", but it usually results in both an increase in overhead [more keyframes] and a decrease in quality [inability to use predictive interpolation, instead relying only on forward-interpolation.]

Mind you, this streaming-enabled encoding is how things were done on the web, before the advent of MPEG-DASH/HLS; and it's still how e.g. the MP2 encoding of digital cable/satellite video works. But we don't really want to go back to those days. They kind of sucked.

Jumping to random byte offsets in a video also tends to screw with any embedded data streams like subtitles or thumbnails, which tend to just be stored in most media container formats as a single chunk at the beginning/end of the file, rather than being spread or copied across the stream. Again, the kind of captioning done back in the MP2 days is immune to this, but it kind of sucked as well (e.g. it wouldn't trigger if you happened to skip to the millisecond after the instruction for it appeared in the stream, often leaving you with ~30 seconds of untranslated audio.)

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by janjanson on Monday May 04, 2020 @09:53PM (#23075543)
I don't think this is quite right. If you serve an mp4 with H.264 video statically on any basic webserver it will just work in browser through plain HTTP, without need need for MPEG-DASH/HLS. Every widely used mediaplayer/browser just downloads the nearest chunk (keyframe point) behind the time that was seeked to can resume the playback from there. This point is found through an index stored in a container format. For basically every video format these days (say, at least as new as H.264), regular settings make this only a few more seconds of video to download and decode before the seeked point and basically happens instantly for normal online consumption. In H.264 forward prediction (through two-pass encoding) will playback fine too.

I think what you're saying applies more to a setting where the video is being streamed live, so that you cannot access the start of the file to get keyframe metadata. In that case HLS and MPEG-DASH help.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by anderspitman on Monday May 04, 2020 @04:44PM (#23073024)
That's odd. I can't recall ever having a problem with browsers playing mp4s in a vanilla <video> tag, as long as I encode them in the main h264 profile, AAC audio, and MOOV atom at the front (see [0] for ffmpeg command). Obviously the server has to support Range: byte requests.

My impression is DASH/HLS are mostly useful for adjusting bitrate on the fly.

[0]: https://superuser.com/a/438471/402047

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by sp332 on Monday May 04, 2020 @05:05PM (#23073243)
When the parent talks about jumping to random byte offsets, they mean you don't have the first part of the file at all. You just have an arbitrary 512-MB chunk out of the middle.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Dylan16807 on Monday May 04, 2020 @07:27PM (#23074590)
But they were claiming that a single monolithic file would break too, which is not the case. The browser does a range request to get the first part, then a range request to get the part you're playing, and it works.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by derefr on Monday May 04, 2020 @06:11PM (#23073971)
Right. Like tuning into a digital-cable signal "in the middle"†. You just get bytes of the stream starting from an arbitrary offset, without having seen/processed anything before that (and without even being able to request anything before that), and you need to resynchronize from what you've got.

† I mean, a digital-cable video stream is always "in the middle" unless you're just starting a VOD stream, but still.

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by Dylan16807 on Monday May 04, 2020 @07:22PM (#23074545)
The browser just reads the TOC-ish stuff from the first chunk. Trust me, it works. I regularly load plain old multi-hundred-megabyte mp4s in my browser, off a web server, and skip around without problems. The default keyframe interval from x264 is fine. You don't have to do any horrible things to the encoding, you just have to start loading a few seconds before the seek point. Which the browser does automatically.

Do browsers even support embedded subtitles?

Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by shockinglytrue on Monday May 04, 2020 @12:53PM (#23070383)
CloudFlare's free plan can and does end at any moment, I wouldn't rely on it for any serious application
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by hartator on Monday May 04, 2020 @12:40PM (#23070185)
Curious as well.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by rochester46 on Monday May 04, 2020 @01:15PM (#23070672)
Does AWS (and GCP and Azure) not do the exact same thing when transferring data out via Cloudfront (or their respective CDN)? Is Backblaze really special in that regard?
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by budmang on Monday May 04, 2020 @01:47PM (#23071062)
(backblaze ceo here) AWS is free to Cloudfront because that's an AWS service. AWS/GCP/Azure are not free (and quite expensive) to Cloudflare. Backblaze is free to Cloudflare.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by microcolonel on Monday May 04, 2020 @03:44PM (#23072435)
And Cloudflare is also the winning reverse proxy caching provider, the one that people tend to use even despite transit charges to AWS/GCP/Azure.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by bvivek77 on Tuesday May 05, 2020 @02:33AM (#23077139)
Bit confusing. Need more clarity. Read 10 times. Pls recheck mr. ceo :)
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by wrkronmiller on Monday May 04, 2020 @01:21PM (#23070742)
Cloudfront costs a non-trivial amount of money per GB: https://aws.amazon.com/cloudfront/pricing/
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by rochester46 on Monday May 04, 2020 @01:29PM (#23070858)
How does that compare to Cloudflare? Is Cloudflare completely free for CDN? I can't find a clear pricing page.
Re:Backblaze B2 Cloud Storage Now Has S3 Compatible APIs by ksec on Monday May 04, 2020 @03:14PM (#23072088)
https://www.cloudflare.com/plans/

Cloudflare dont charge per GB. They are more like per "feature" CDN.