News for hackers. Stuff that matters.
Re:Google loses data as lightning strikes by mathattack on Wednesday August 19, 2015 @11:45AM (#10086299)
Beyond security, this highlights one of the main issues with the cloud. Was there no backup?

Of course once you get beyond the headline, I think most people are much worse with protecting themselves from rare outages than Google.

Re:Google loses data as lightning strikes by bsurmanski on Wednesday August 19, 2015 @11:53AM (#10086371)
As mentioned in the article, it only affected recently written data of 'Google Compute Engine' services. GCE allows user to launch VMs and generate arbitrary data on the server.

Normally, Google redundantly distributes out data to at least 3 different geographically distinct locations. Check out the 'BigTable' white paper [0] for more info.

For 99% of cases (and pretty well all user cases), this would not cause data lose. The key here is that the data was generated on the servers and did not have a chance to duplicate before the event.

[0] http://research.google.com/archive/bigtable-osdi06.pdf

Re:Google loses data as lightning strikes by axiak on Wednesday August 19, 2015 @12:39PM (#10086709)
Nowadays they use Reed–Solomon coding to effectively distribute their data without copying it to 3 places.
Re:Google loses data as lightning strikes by Artemis2 on Wednesday August 19, 2015 @12:49PM (#10086775)
Do you have a source for that? Why would they start using it now?
Re:Google loses data as lightning strikes by azurezyq on Wednesday August 19, 2015 @02:11PM (#10087351)
Disclaimer: googler, but not working on storage.

link here: http://static.googleusercontent.com/external_content/untrust...

actually in colossus one can tune RS coding parameters per file, to get a tradeoff between performance/durablity.

RS coding uses less copies, but same level of safety (tradeoff is the recovery computation time.)

Re:Google loses data as lightning strikes by magicalist on Wednesday August 19, 2015 @01:25PM (#10087033)
Also how would that help with geographic redundancy? Local recovery from errors does you no good if your datacenter gets wiped out in a flood.
Re:Google loses data as lightning strikes by axiak on Wednesday August 19, 2015 @04:35PM (#10088264)
I can't find the presentation anywhere, but at HBaseCon 2014 one of the lead developers of Bigtable stated that they went to RS. Even for their older databases.

EDIT: In this video https://vimeo.com/100153741, around the 23 minute mark

Re:Google loses data as lightning strikes by ismavis on Wednesday August 19, 2015 @11:50AM (#10086344)
You find more details in the Google incident report: https://status.cloud.google.com/incident/compute/15056#57195...
Re:Google loses data as lightning strikes by gtaylor on Wednesday August 19, 2015 @12:18PM (#10086569)
> Beyond security, this highlights one of the main issues with the cloud. Was there no backup?

Unless I am specifically paying for backup service, I wouldn't expect/want them to do that for me. And even if I was, I'd still have offsite backups if the system was important.

If you aren't being responsible with your backups in a noisy environment like Google Cloud/AWS, understand that you are vulnerable to freak accidents like this. Google/AWS's job in all of this is to try to reduce the frequency of issues and to minimize the impact.

Re:Google loses data as lightning strikes by MichaelGG on Wednesday August 19, 2015 @12:58PM (#10086834)
If you were running the data center instead of Google, it'd suddenly stop being cloud. How would that change lightning and power loss?

Google Compute Engine offers customers the option to make snapshots for backup, or use a true "cloud" storage engine. If anyone lost data here it was customers explicitly not doing backups and only using a single zone. I don't know why anyone would expect different. GCE easily allows you to network machines in multiple data centers, but close geographically. So you'd only need to handle region-wide disasters.

Re:Google loses data as lightning strikes by mankyd on Wednesday August 19, 2015 @11:49AM (#10086335)
The article states that it was "some recently written data [...] located on storage systems which were more susceptible to power failure". Perhaps it hadn't made it to a backup.
Re:Google loses data as lightning strikes by sowbug on Wednesday August 19, 2015 @12:48PM (#10086768)
This was closer to a colocated server than a cloud service. GCE gives you a VM that you install your favorite OS on. Google keeps the VM running, but the rest (such as whether the server backs itself up) is up to you.