News for hackers. Stuff that matters.
Re:Introducing Google Cloud Storage Nearline by fuzzythinker on Wednesday March 11, 2015 @10:54AM (#9184363)
Unless Glacier just changed their pricing, how is it cheaper and simpler? Glacier is also $.01/GB storage, but only $.09/GB retrieval, which is cheaper than google's $.12/GB. Glacier also comes with 1st GB retrieval free/mo.
Re:Introducing Google Cloud Storage Nearline by panarky on Wednesday March 11, 2015 @11:13AM (#9184466)
Glacier pricing is surprisingly complicated, and the actual cost can be much higher than $0.01 per GB-month if you don't read the fine print.

The biggest gotcha is that you can only access 0.17% of the data you've stored in any given day without extra charges. So if you've stored 1000 GB, you can only access 1.7 GB per day for free.

The cost for going over your daily "retrieval allowance" can be large, because cost is driven by "peak retrieval rate". They find the highest hourly retrieval rate over the entire month, and charge the whole month's retrieval at the peak rate.

This can get expensive fast. Again, with 1000 GB stored, if you retrieve 200 GB over the course of 4 hours, your hourly retrieval rate is 200 GB / 4 hours = 50 GB per hour. Your free retrieval allowance is 1.7 GB per day / 24 hours = 0.07 GB / hour. So your excess retrieval is 49.93 GB per hour.

Amazingly, that 49.93 GB per hour for 4 hours is charged for all 720 hours in the month, so that's 49.93 * 720 * 0.01 = $359.49.

That's an astonishing $1.79 per GB just to retrieve one-fifth of the data from Glacier storage.

So Glacier only makes sense for true cold storage, where you are very unlikely to touch more than 0.00007 of it in any given hour.

Google's Nearline is much simpler and faster for the same storage cost -- and far cheaper if you need to access more than a tiny fraction of the data.

Source: http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...

Re:Introducing Google Cloud Storage Nearline by AceJohnny2 on Wednesday March 11, 2015 @02:07PM (#9185582)
Thanks for the detailed breakdown, that's really interesting! Sounds like you were bitten by this? :)
Re:Introducing Google Cloud Storage Nearline by ak217 on Wednesday March 11, 2015 @03:42PM (#9186534)
Don't know about the OP, but we were bitten by this - badly - after our logic to throttle Glacier retrieval failed.

AWS has now added the ability to set a spending limit to avoid runaway retrievals. Nonetheless, from my point of view the ultra-complicated pricing scheme is nothing short of a disaster for Glacier as a product. I think it will continue to seriously impact its uptake, and Google is smart to exploit the opportunity to offer a simple, straightforward pricing scheme.

Re:Introducing Google Cloud Storage Nearline by NeutronBoy on Wednesday March 11, 2015 @05:10PM (#9187328)
The use case is people who have a requirement to store data and probably never need to access it, either for reg requirements, or legal requirements, etc. In which case paying a couple of hundred bucks for retrieval is fine, especially if they can access at a high level of granularity

edit: Also as others have pointed out, Nearline is limited in retrieval time as well, so the cost difference isn't nearly as large.

Re:Introducing Google Cloud Storage Nearline by Siliticxx on Wednesday March 11, 2015 @02:42PM (#9185953)
Ok. I have to this to this. i'm sorry.

I guess he would say he was... sunglasses FROSTBITTEN YEAH!

Re:Introducing Google Cloud Storage Nearline by AceJohnny2 on Wednesday March 11, 2015 @03:58PM (#9186695)
I understand, but please go stand in the corner there and think about what you've done.
Re:Introducing Google Cloud Storage Nearline by oconnore on Wednesday March 11, 2015 @04:34PM (#9187003)
Why does Amazon charge like this? It seems like on the rare occasion that you need to send some person to go grab the tape/disk from storage and bring it online, Amazon would want you to get all the data you need and put it back in storage.

Incentivizing users to bring the data online once a day to trickle it out seems bad for all involved.

Re:Introducing Google Cloud Storage Nearline by tveita on Wednesday March 11, 2015 @05:57PM (#9187673)
I think it makes most sense to think of it as paying for access to the tape robot.

If Amazon store your data in a tape archive (I don't know if they do, but they at least seem to have similar constraints), they can only access a small portion of the stored data at a time, so they need to control how often people request data.

They could just rate limit everyone, but this way allows people to pay for priority in an emergency while still discouraging everyday read requests.

The pricing makes more sense if you're a large user with data spanning several tapes than if you're in the single terabyte range, but the low limit still discourages you from making requests causally, which helps them keep their SLA.

If they predict that you'll trickle out your small file, they can just read out everything on the first access and cache it online, so there's no extra trips to the archive for them.

Re:Introducing Google Cloud Storage Nearline by breakingcups on Thursday March 12, 2015 @05:17AM (#9189725)
Note: The following is speculation best I know. I'm recalling from memory what I've read on the internet written by someone who did not have a direct source.

Glacier uses low-speed (5400 RPM) consumer drives, which they then clock down lower to save energy. Any given 'rack' only has enough power to power a few drives on that rack, the rest is powered down.

To prevent multiple customers from all trying to pull their data out they needed to introduce a rate limiting system, which they did with this exorbitant pricing.

Re:Introducing Google Cloud Storage Nearline by discodave on Wednesday March 11, 2015 @06:09PM (#9187749)
You assume Glacier uses tape storage... But the same principle still applies, they're trying to prevent people clogging up the network so when somebody does need to do an emergency restore there is lots of spare network capacity.
Re:Introducing Google Cloud Storage Nearline by kryps on Wednesday March 11, 2015 @11:12AM (#9184463)
Except for the Glacier retrieval fees which can be insane if you choose to retrieve more than 0.17% of your stored data per day.

https://aws.amazon.com/glacier/faqs/#How_much_data_can_I_ret...

Re:Introducing Google Cloud Storage Nearline by t0mas88 on Wednesday March 11, 2015 @03:48PM (#9186596)
But if you retrieve the same 1000 GB from Glacier in the same 3 days that Google takes you would pay a 98 dollar retrieval fee. And Amazon would charge 300 dollar less traffic cost ($ 0.09 vs $ 0.12 / GB for 1000 GB) so in the end Glacier is still 202 dollar cheaper AND allows you to retrieve the data much faster if you really have to at a certain price.

So Google looks cheaper/simpler, but the amazon model is actually quite good for disaster recovery where you may be in a situation where you "need it now at any cost"

Re:Introducing Google Cloud Storage Nearline by panarky on Wednesday March 11, 2015 @03:54PM (#9186658)
Internet egress for 1000 GB is $120 for Google and $90 for Amazon, a difference of $30 not $300.
Re:Introducing Google Cloud Storage Nearline by omh on Wednesday March 11, 2015 @11:07AM (#9184432)
Glacier have additional fees depending on how fast you retrieve the data. Honestly I'm not sure what they are because I've never had to use it, but my impression is that these fees could be quite high.
Re:Introducing Google Cloud Storage Nearline by Twirrim on Wednesday March 11, 2015 @11:35AM (#9184593)
They can be. It's possible to set a retrieval policy on your bucket either at free tier, or a fixed price point so you can control the cost of your final bill if you want to retrieve faster than the free tier
Re:Introducing Google Cloud Storage Nearline by keysheath on Wednesday March 11, 2015 @11:13AM (#9184473)
At least with Glacier I get the option of retrieving faster. Nearline is going to take more than 3 DAYS to retrieve my data if I only have 1TB stored (3MB/s).
Re:Introducing Google Cloud Storage Nearline by panarky on Wednesday March 11, 2015 @11:28AM (#9184558)
Let's assume that with Glacier you can retrieve your 1000 GB in 4 hours instead of 3 days.

You will be surprised by a retrieval fee of 1000 / 4 * 720 * 0.01 = $1,800.

If you want to get your 1000 GB for free, you'd have to request no more than 69 MB per hour. It will take you about 20 months to get your data!

With Glacier, limit retrievals to less than 5% per month / 30 days / 24 hours = 0.007% of total data stored per hour to avoid these ugly surprises.

Source: http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...

Re:Introducing Google Cloud Storage Nearline by moe on Wednesday March 11, 2015 @11:32AM (#9184579)
For many people waiting 3 days is more attractive than paying the ~$1000 that it would cost to retrieve it in one day from Glacier.
Re:Introducing Google Cloud Storage Nearline by keysheath on Wednesday March 11, 2015 @11:44AM (#9184646)
For disaster recovery 3 days might mean the end of your business.
Re:Introducing Google Cloud Storage Nearline by jacquesm on Wednesday March 11, 2015 @12:44PM (#9184994)
For disaster recovery $1000 is peanuts, that's a company thing, GP mentioned 'people', not companies, different use-case.
Re:Introducing Google Cloud Storage Nearline by prirun on Thursday March 12, 2015 @10:40AM (#9191061)
I wrote a backup program that uses Glacier, and the retrieval policies are nearly impossible to manage and explain. But the thing I really don't get is that you can use Amazon Import to read a hard drive into Glacier, but you can't use Amazon Export to get a drive full of data back out. You can with S3.

In a disaster situation, a company is going to want hard drives sent to them next day. As others have mentioned, this isn't a money thing, it's a time issue. But it isn't available with Glacier (probably not with Google either...)

Re:Introducing Google Cloud Storage Nearline by relate on Wednesday March 11, 2015 @11:56AM (#9184717)
If you're willing to pay more for faster retrieval I guess you could store e.g. 5TB of random data to get a fast "base speed"- cutting those 3 days down to 12 hours.
Re:Introducing Google Cloud Storage Nearline by dragonwriter on Wednesday March 11, 2015 @12:58PM (#9185078)
Or use Nearline for archival data as well as disaster recovery. You won't need to transfer the archival data for normal disaster recovery scenarios, so it'll be -- for that scenario -- extra data that boosts your speed, but it's still, on its own, a useful use of the service.
Re:Introducing Google Cloud Storage Nearline by teraflop on Wednesday March 11, 2015 @12:51PM (#9185041)
That would be more expensive than just storing 1TB with the normal, full speed Cloud Storage service.
Re:Introducing Google Cloud Storage Nearline by rsync on Wednesday March 11, 2015 @03:28PM (#9186368)
"Unless Glacier just changed their pricing, how is it cheaper and simpler? Glacier is also $.01/GB storage, but only $.09/GB retrieval, which is cheaper than google's $.12/GB. Glacier also comes with 1st GB retrieval free/mo."

As long as we are comparing effective pricing, which is the only number that matters with Glacier and "Google Nearline" (and, to some degree, S3) it should be noted that rsync.net PB-scale is 3.0 cents with no additional charges.

That is, the effective price, no matter what your use-case or traffic/usage, is 3.0 cents per GB.[1]

Of course, you do have to buy a petabyte of it ...

[1] http://www.rsync.net/products/petabyte.html

Re:Introducing Google Cloud Storage Nearline by jhgg on Wednesday March 11, 2015 @11:32PM (#9189085)
Not to nitpick, but your CEO page is a bit misleading.

    > "a two year contract is required." [1]
    > "There are no contracts, overages, fees, or license charges at rsync.net." [2]
[1] http://www.rsync.net/products/petabyte.html

[2] http://rsync.net/products/ceopage.html