News for hackers. Stuff that matters.
Re:Why you should not use Google Cloud by markbnj on Saturday June 30, 2018 @02:01PM (#17431879)
I can't speak to the specific incident. We've been running almost 400 servers (instances and k8s cluster nodes) for over a year on GCP and we've been quite happy with the performance and reliability, as well as the support response when we have needed it. I did want to address this comment...

> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.

You should never be in this position. If this were to happen to us we would be able to create a new project with a different payment instrument, and provision it from the ground up with terraform, puppet and helm scripts. The only thing we would have to fix up manually are some DNS records and we could probably have everything back up in a few hours. Eventually when we have moved all of our services to k8s I would expect to be able to do this even on a different cloud provider if that were necessary.

Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @02:06PM (#17431904)
Restarting the service from scratch is one thing, but what about all your data? Some of these services have 100's T's of data hanging of them and if Google would delete that because of some perceived violation of their terms then that is not something you can recover from in a couple of hours, if at all.

This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account. That protects against all kinds of accidents but also against malice.

Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @02:26PM (#17432014)
Backup is a thing. If your company is making millions of dollars off your business you should have a redundant backup of everything including (especially) your data.
Re:Why you should not use Google Cloud by nemothekid on Saturday June 30, 2018 @03:45PM (#17432417)
Yes, we have backups. The problem with `data` is not the fact I have backups, is that at a certain scale I will have so much data that "moving providers" could take on the order weeks.

If OP happened to me, sure yes I could have my entire infra on AWS/Azure/whatever else Terraform supports in an hour, maybe more to replace some of the tiny cloud specific features we use. But if it takes me a day to me to just move the data into Azure, thats an entire lost business day of productivity.

Re:Why you should not use Google Cloud by southerndrift on Saturday June 30, 2018 @05:25PM (#17432837)
If it takes weeks then you should choose a second provider where you can show up with your backup hard drives or whatever you use and plug them in. Moving data physically is an option.
Re:Why you should not use Google Cloud by rhizome on Saturday June 30, 2018 @05:51PM (#17432917)
"Never underestimate the bandwidth of a semi full of harddrives."
Re:Why you should not use Google Cloud by mooreds on Sunday July 01, 2018 @01:24PM (#17436501)
Re:Why you should not use Google Cloud by markbnj on Saturday June 30, 2018 @04:08PM (#17432530)
Note that I did not imply that restoring the service to a different project or provider would always be easy or fast (certainly in the case of very large data volumes it would be neither of those things). I was addressing the prospect of losing "years of work" as was stated in the OP. That sort of implies that most or all of what they did over that time is recorded only in the current state of the GCP project that was disabled, and that is a really terrifying position to be in.
Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @04:08PM (#17432529)
People usually end up using backup or move infrastructure on short notice in catastrophic situations, which is presumably rare. Days worth of work to bring back your business in catastrophic downtime - doesn't seem like a bad thing at all to me. If anything, it sounds like a very well organized development flow with very optimistic time-frame.
Re:Why you should not use Google Cloud by csydas on Saturday June 30, 2018 @03:02PM (#17432208)
This vastly simplifies the situation, especially when the cloud is involved. Having a backup, much less a replica of such data requires an enormous infrastructure cost, whether it's your own or someone else's infrastructure. The time to bring that data back to a live and stable state again also is quite costly. (note the stable part)

It's a simple truth that even if you are at the millions of dollars point, there is a data size at which you are basically all-in with whatever solution you've chosen, and having a secondary site even for a billion dollar company can be exceptionally difficult and cost prohibitive to move that sort of data around, again especially when you're heavily dependent on a specific service provider.

Yes, the blame in part lies with making the decision to rely on such a provider. At the same time, there are compelling arguments for using an existing infrastructure instead of working on the upkeep of your own for data and compute time at that scale. Redundancy is built into such infrastructures, and perhaps it should take a little more evidence for the provider to decide to kill access to everything without hard and reviewed evidence.

Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @03:14PM (#17432269)
It might be too expensive for some people. But really there is no other solution other than full backup of everything. Relying on a single point of failure, even on an infrastructure with a stellar record, is just a dead man walking.
Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @04:11PM (#17432540)
And then of course there is the important bit that from a regulatory perspective 'just a backup' may be enough to be able to make some statements about the past but it won't get you out of the situation where due to your systems being down you weren't ingesting real-time date during the gap. And for many purposes that makes your carefully made back-up not quite worthless but close to it.

So then you're going to have to look into realtime replication to a completely different infrastructure and if you ever lose either one then you're immediately on very thin ice.

It's like dealing with RAID5 on arrays with lots of very large hard drives.

Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @04:39PM (#17432641)
About ~6 years ago, I was involved in a project where data would increase by 100gb per day and the database would also significantly change every day. I vaguely remember having some kind of cron bash script with mysqldump and rsync that would have a near identical offsite backup of data (also had daily, monthly snapshots). We also had a near identical staging setup of our original production application which we would use to restore our application from the near-realtime backup we had running. We had to test this setup every other month - it was an annoying thing to do at first. But we were exceedingly good at it over time. Thankfully we never had to use our backup, but we slept at night peacefully.

Backup is a bit of an art in itself, everyone has a different type of backup requirement for their application, some solutions might not be even financially feasible. You might never end up using your backup ever at all, but all it needs is one very bad day. And if your data is important enough, you will need to do everything possible to avoid that possible bad day.

Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @04:49PM (#17432668)
That's a good scheme. Note how things like GCP make it harder rather than easier to set something like that up, you'd almost have to stream your data in real time to two locations rather than to bring it in to GCP first and then to stream it back out to your backup location.

> Backup is a bit of an art in itself

Fully agreed on that, and what is also an art is to spot those nasty little single points of failure that can kill an otherwise viable business. Just thinking about contingency planning makes you look at a business with different eyes.

Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @02:49PM (#17432150)
Yes, I'm aware of that. But you'd be surprised how many businesses are under the impression that using 'the cloud' obviates the needs for backups. Especially if their data is in the 100's of terabytes.
Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @03:04PM (#17432218)
Non-technical owners making faulty assumptions is not the fault of "Cloud" providers. It's probably common (I faced it myself personally, in a non-cloud situation), but there is nothing the providers can do about unprepared users.
Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @04:21PM (#17432576)
> but there is nothing the providers can do about unprepared users.

That's true, but they can do something to avoid making things worse, see the linked article.

Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @04:53PM (#17432676)
While true. I was specifically referring to this part:

> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.

The comment suggests they are using personal GCP account instead of enterprise account.

Millions of dollars worth of work + imply no backup + non-enterprise account (but expecting enterprise support) + not having multiple forms of payment available.

Combining all these together, it seems like all sorts of things are going wrong here.

I have never used GCP (or any of the big three cloud providers), so I don't know how they are in general, but in this specific case there seems to be faulty planning on the user end.

Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @04:58PM (#17432696)
Agreed, that wasn't smart. But, to their defense, this is how these things start out, small enough to be useful, and by the time they get business critical nobody realizes the silly credit card is all that stands between them and unemployment.
Re:Why you should not use Google Cloud by Endy on Saturday June 30, 2018 @02:28PM (#17432022)
Why does it matter whether you're making millions of dollars? If you have any information which you would like to not lose for any reason, back it up in as many formats and locations as is feasible.
Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @02:57PM (#17432190)
Agreed. I mentioned the money angle because I felt the person I replied to implied that 100s of terabytes of data are too expensive to backup.

If you are making money or if the data is important for you to lose than you should have a backup, anything else is faulty planning.

Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @03:03PM (#17432216)
> I felt the person I replied to implied that 100s of terabytes of data are too expensive to backup.

Well, you felt wrong. Of course you should back up those 100s of terabytes, in fact that it is that much information is an excellent reason on top of all the other ones to back it up, re-creating it is going to be next to impossible.

It's just that the companies I look at - not all, but definitely some - seem to be under the impression that the cloud (or their cloud provider) can be trusted. Which is wrong for many reasons, not just this article.

Re:Why you should not use Google Cloud by Endy on Saturday June 30, 2018 @07:39PM (#17433287)
I forget where I first saw this quoted, but it's relevant here: "There is no 'cloud', only someone else's computer". That's part of why I store very little data online, compared to most people (or the data I actually have/want). Anything I'm not okay with someone else having on their computer is backed up and stored on hard physical media. No cloud provider can be trusted - the moment the government wants in, they'll get in; and the moment it's considered more profitable for the provider to quietly snoop in your stored data, rest assured that they will.
Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @03:08PM (#17432240)
Sorry, I stand corrected.
Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @04:24PM (#17432589)
No problem, it's just that with 'This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account.' that passage I thought I had the backup angle more than covered.

What bugs me about it is that there are some companies that give serious pushback because their cloud providers keep on hammering in to them how reliable their cloud is and that any back-up will surely be less reliable than their cloud solution and oh by the way we also have a backup feature that you can use.

They don't realize that even then they still have all their eggs in the one basket: their cloud account.

Re:Why you should not use Google Cloud by pavs on Saturday June 30, 2018 @05:16PM (#17432802)
It's strange, but I completely missed the last part about backup from your comment. I have no idea how I missed it. Had I seen it would make my comment redundant and I would have never replied at all.

I only saw that part of the comment much much later.

Re:Why you should not use Google Cloud by jacquesm on Saturday June 30, 2018 @05:52PM (#17432922)
Well, it definitely wasn't added in a later edit, or at least, not that I'm aware of, though I do have a tendency to write my comments out in bits submitted piece-by-piece. Even so, I wouldn't worry about it, I tend to miss whole blocks of text with alarming regularity while reading through stacks of pdfs and when comparing notes with colleagues we always wonder if we've been reading the same documents (they have the same problem...). Reading in parallel is our way of trying to ensure we don't miss anything and unfortunately it is not a luxury.

Often the effects are more subtle, reading what you think something said rather than what it actually said, or missing a negation or some sub-clause that materially alters the meaning of a sentence.

Even in proofreading we find stuff that is so dead obvious it is embarrassing. On the whole visual input for data is rather unreliable, even when reading stuff you wrote yourself, which I find the most surprising bit of all.

Studying this is interesting, and to some extent important to us due to the nature of our business, missing critical info supplied by a party we are looking at could cause real problems so we have tried to build a process to minimize the incidence of such faults, even so I'm 100% sure that with every job we will always miss something, and I live in perpetual fear of that something being something important.

Re:Why you should not use Google Cloud by ma2rten on Saturday June 30, 2018 @02:09PM (#17431914)
Huh? You can provision new servers, but you can't just easily move over all the data, can you?
Re:Why you should not use Google Cloud by lotyrin on Saturday June 30, 2018 @02:52PM (#17432164)
Why not? You should have backup strategy with business-acceptable RPO/RTO.
Re:Why you should not use Google Cloud by markbnj on Saturday June 30, 2018 @02:30PM (#17432028)
Easily? That's in the eye of the person doing the work I guess. But we have backups and could restore our databases.
Re:Why you should not use Google Cloud by deltateam on Saturday June 30, 2018 @02:05PM (#17431898)
I wrote a similar comment, thats what the best practices are for