News for hackers. Stuff that matters.
Re:DigitalOcean lost our server by gdgtfiend on Thursday February 04, 2016 @03:13PM (#11036814)
This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"
Re:DigitalOcean lost our server by theptip on Thursday February 04, 2016 @03:36PM (#11037000)
Exactly, to use a now-trite metaphor: your cloud instances are cattle, not pets.
Re:DigitalOcean lost our server by endymi0n on Thursday February 04, 2016 @04:15PM (#11037280)
While my first intuition was to agree with you, there's certainly an upcoming generation of developers who have never operated their own root servers and the abstraction level in the cloud nowadays is so high it makes you easily forget that "Droplets" are just VMs are just servers running software. On the other hand, hardware reliability has increased in recent years due to RAID, fully redundant networking & power adaptors, you name it.

These facts combined certainly make it easy for the younger generation to forget that redundancy and backup still don't replace each other (never did).

Re:DigitalOcean lost our server by beat on Thursday February 04, 2016 @04:21PM (#11037330)
Well, there's one way to teach these kids...

(Could be worse. Their first lessons could be like mine, in C! If anything will make you paranoid, writing C will.)

Re:DigitalOcean lost our server by XorNot on Friday February 05, 2016 @03:20AM (#11040255)
This seems like the wrong lesson is being taught though somewhere - the delete button is right there. It should be much more obvious that is very easy to make a server go away.
Re:DigitalOcean lost our server by cortesoft on Thursday February 04, 2016 @03:53PM (#11037131)
It is no different if you are running your own hardware - your server hardware can fail no matter who owns the machine.
Re:DigitalOcean lost our server by techjuice on Thursday February 04, 2016 @05:38PM (#11037857)
This is so true, I wish there was a way to make this more noticeable (flashing red lights on the order forms, regular email warnings about the single point of failure,etc.). This simple fact is taught in entry level system administration courses (RAID, Disaster Recovery Plans, Replication, High Availability, Fault Tolerance, etc.) but in reality some seem to not plan for it when moving to cloud services. I always suggest people set their apps up to be able to run out of multiple availability zones in case one goes down if it is very important to them. For those that are not able to I will suggest they at least setup replication to another server within the same data center and an offsite location in case their server has hardware failure, an account gets locked or other possible situation to help take of the just in case scenarios that occur to everyone.
Re:DigitalOcean lost our server by lotyrin on Thursday February 04, 2016 @04:50PM (#11037557)
Yep, somehow cloud customers have decided that by being in the cloud everything is redundant and durable and has several other magical properties.

At one of my positions we had stupidly put too many eggs in the basket of a single physical machine. Its disk controller failed in a way that it trashed the data volumes. I was unable to convince anyone that "move to Amazon" was not a one-step solution to "how do we make sure this never happens again".

Re:DigitalOcean lost our server by moonchrome on Thursday February 04, 2016 @08:56PM (#11038864)
>somehow cloud customers have decided that by being in the cloud everything is redundant and durable and has several other magical properties

To be fair the point of the cloud is that they deal with redundancy, HA, distributing to multiple datacenters, etc. through their services - but you need to use and understand implications of said services to leverage that.

Re:DigitalOcean lost our server by lotyrin on Friday February 05, 2016 @04:57PM (#11044957)
When cloud == SaaS, sometimes. When cloud == IaaS, nope.

But that doesn't stop the misconception.

Re:DigitalOcean lost our server by Dylan16807 on Thursday February 04, 2016 @06:27PM (#11038105)
If your server does something weird and corrupts the filesystem, you still have access to it. That can be a huge difference when it comes to restoring the most recent data.
Re:DigitalOcean lost our server by nickbauman on Thursday February 04, 2016 @04:28PM (#11037392)
CAP theorem applies everywhere, but most software practices still do not assume that you must code for your app to be able to handle performance variation and partial failure state both of which are common scenarios of cloud infra.
Re:DigitalOcean lost our server by bluedino on Thursday February 04, 2016 @03:18PM (#11036858)
Remember the blog posts during the early days of AWS? People didn't realize their new elastic cloud servers could go into thin air
Re:DigitalOcean lost our server by Piskvorrr on Friday February 05, 2016 @04:15AM (#11040407)
Dissolve into thin air? That's what clouds do, right? (One woulda thunk the word "cloud" in itself would be warning aplenty ;))
Re:DigitalOcean lost our server by Fizzadar on Thursday February 04, 2016 @06:42PM (#11038180)
I have to disagree - DO's "cloud servers" are equivalent to virtual private servers, which you would never expect to loose in this manner from other providers.

The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".

Re:DigitalOcean lost our server by RubyPinch on Thursday February 04, 2016 @08:00PM (#11038627)
I'd expect to lose them.

Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.

Re:DigitalOcean lost our server by e12e on Thursday February 04, 2016 @09:19PM (#11038958)
I remember a ~36 hour down time with a serious uk provider (in my case just a shell account, but they also housed managed and unmanaged servers). They had redundant fiber links to the NOC, from competing providers. But turns out they both ran through the same box under a high way. And then someone blew that box up in order to knock out alarms while they committed a robbery...
Re:DigitalOcean lost our server by speeder on Friday February 05, 2016 @01:59AM (#11040015)
My family business website got killed once by a flood.

And many years ago it suffered a truck crash ( the truck crashed on the lamp post right outside the server company building and destroyed the telecom-related wires)

Re:DigitalOcean lost our server by bitshepherd on Thursday February 04, 2016 @06:56PM (#11038267)
Welcome to "The Cloud". VPS can be effectively rebranded as "cloud", except when a server has problems you shoot first and ask later.
Re:DigitalOcean lost our server by __michaelg on Thursday February 04, 2016 @03:34PM (#11036985)
Agreed. This shouldn't be an issue for anyone using cloud services -- or any computer really. Except for some super duper redundant mainframes maybe, you should always be able to deal with losing a component not matter what it is. If you really care, always have multiple instances, in multiple (availability) zones, etc.
Re:DigitalOcean lost our server by nailer on Thursday February 04, 2016 @04:27PM (#11037382)
> You should be designing for failure in "the cloud"

DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

Re:DigitalOcean lost our server by vardump on Thursday February 04, 2016 @04:33PM (#11037429)
That's a very unreasonable expectation at cloud price levels.

Does any cloud provider have SAN backed VMs? If any do, at what price?

Re:DigitalOcean lost our server by hyperknot on Thursday February 04, 2016 @04:52PM (#11037569)
Not a huge company like DO, but iwStack [1] provides a SAN backed cloud, with selectable KVM/XEN instances, custom ISO and virtual network support. The prices are similar to DO. [1] http://iwstack.com/
Re:DigitalOcean lost our server by vardump on Thursday February 04, 2016 @05:02PM (#11037654)
That's pretty damn interesting for more traditional hosting type usage.

Seems like they sell 50 GB of real SAN backed storage for just 3.6€ per month.

How can they afford to be so cheap?

Re:DigitalOcean lost our server by hyperknot on Friday February 05, 2016 @06:38AM (#11040809)
I think one of the reasons is that they only have a small number of datacenters, have a small number of (very friendly) staff and are not going after the mass market like DO, and I guess they don't spend anything on marketing.
Re:DigitalOcean lost our server by mappu on Thursday February 04, 2016 @06:11PM (#11038027)
That's not even particularly cheap per-GB - BuyVM ( http://buyvm.net/storage-vps/ ) will sell you 250GB for 7USD/mo.

Disclaimer: happy customer

Re:DigitalOcean lost our server by vardump on Thursday February 04, 2016 @06:23PM (#11038084)
> That's not even particularly cheap per-GB - BuyVM ( http://buyvm.net/storage-vps/ ) will sell you 250GB for 7USD/mo.

Except that is not SAN backed storage.

From the website:

> We only use RAID-60 drive arrays with a minimum of 16 drives

That's somewhat scary. RAID-6 might be almost ok, but striped? No thank you. I bet they don't also have block level checksums.

Re:DigitalOcean lost our server by derefr on Thursday February 04, 2016 @04:48PM (#11037542)
That's what AWS's "Elastic Block Storage" is. You can turn it off and just use instance storage (and I personally prefer to, for truly ephemeral nodes), but it increases spawn time since your disk image actually has to get copied over to the VM host machine in that case, rather than just "attached" over EBS.
Re:DigitalOcean lost our server by vardump on Thursday February 04, 2016 @04:52PM (#11037574)
Then why EBS failure rate is several orders of magnitude higher than in SAN deployments? A SAN provider would be quickly out of business with 0.1-0.5% annual failure rate.

SAN reliability ratings start at 99.999%.

Re:DigitalOcean lost our server by derefr on Thursday February 04, 2016 @05:04PM (#11037666)
Just because it's a SAN doesn't mean a given abstract block device from it is backed by RAID. It's literally just a multiplexed and QoSed network-attached storage cluster.

I actually prefer the lower-level abstraction: if you want a lower failure rate (or higher speed), you can RAID together attached EBS volumes yourself on the client side and work with the resultant logical volume.

Re:DigitalOcean lost our server by prodigal_erik on Thursday February 04, 2016 @05:01PM (#11037644)
On AWS, an EBS volume is only usable from one availability zone. You still need to use application-level replication to get geographic redundancy for important data, and when you have that, EBS just lets you be lazy rather than eager about copying a snapshot to local instances.
Re:DigitalOcean lost our server by derefr on Thursday February 04, 2016 @07:03PM (#11038305)
I guess I was thinking in terms of using EBS for ephemeral business-tier nodes, rather than as the backing store of your custom database-tier. (I usually use AWS's RDS Postgres for my database.)

For ephemeral business-tier nodes, EBS gives you a few advantages, but none of them are that astounding:

• the ability to "scale hot" by "pausing" (i.e. powering off) the instances you aren't using rather than terminating them, then un-pausing them when you need them again;

• the ability for EC2 to move your instances between VM hosts when Xen maintenance needs to be done, rather than forcibly terminating them. (Which only really matters if you've got circuit-switched connections without auto-reconnect—the same kind of systems where you'd be forced into doing e.g. Erlang hot-upgrades.)

• the ability to RAID0 EBS volumes together to get more IOPS, unlike instance storage. (But that isn't an inherent property of EBS being network-attached; it's just a property of EBS providing bus bandwidth that scales with the number of volumes attached, where the instance storage is just regular logical volumes that all probably sit on the same local VM host disk. A different host could get the same effect by allocating users isolated local physical disks per instance, such that attaching two volumes gives you two real PVs to RAID.)

• the ability to quickly attach and detach volumes containing large datasets, allowing you to zero-copy "pass" a data set between instances. Anything that can be done with Docker "data volumes" can be done with EBS volumes too. You can create a processing pipeline where each stage is represented as a pre-made AMI, where each VM is spawned in turn with the same "working state" EBS volume attached; modifies it; and then terminates. Alternately, you can have an EC2 instance that attaches, modifies, and detaches a thousand EBS volumes in turn. (I think this is how Amazon expected people would use AWS originally—the AMI+EBS abstractions, as designed, are extremely amenable to being used in the way most people use Docker images and data-volumes. The "AMI marketplace" makes perfect sense when you imagine Docker images in place of AMIs, too. Amazon just didn't consider that the cost for running complete OS VMs, and storing complete OS boot volumes, might be too high to facilitate that approach very well. Unikernels might bring this back, though.)

Re:DigitalOcean lost our server by tbyehl on Thursday February 04, 2016 @09:22PM (#11038978)
SANs aren't magical boxes incapable of losing data.
Re:DigitalOcean lost our server by mrweasel on Friday February 05, 2016 @02:57AM (#11040184)
I worked for a company that believed that, for a while.

They bought a hugely expensive SAN solution (HP I think). One of my questions was: What happens when the SAN fails? Well you see, because it has redundancy, that can't happen. Clever uh?.

First time it failed, everything gridded to a halt for two days. The second time they did better and had it running within eight hours.

Re:DigitalOcean lost our server by nailer on Friday February 05, 2016 @03:13AM (#11040238)
Nobody claimed that.
Re:DigitalOcean lost our server by tbyehl on Friday February 05, 2016 @10:56AM (#11042202)
Yet you suggest DO should be using SANs when there isn't enough information provided to know if it would have made a difference in the outcome.
Re:DigitalOcean lost our server by nailer on Friday February 05, 2016 @12:52PM (#11043182)
I suggest Digital Ocean should be using any mechanism that stops a single server from affecting data. A SAN backed block store allows you to provide rules for the level of redundancy required, as well as replication mechanisms for things like firmware failures, to a much greater degree than a host backed RAID array.

That's completely different from suggesting 'SANs are magical boxes incapable of losing data'. Have you considered apologising?

Re:DigitalOcean lost our server by tbyehl on Friday February 05, 2016 @01:05PM (#11043295)
Servers affect data. That's kind of the whole point of servers. Maybe a multi-SAN setup with synchronous replication would have prevented the data loss. Maybe not. Not enough information has been provided to know if the cause of the data loss was the storage or the server.

But if DO had a multi-SAN setup with synchronous replication this thread wouldn't exist because DO's business model of "super cheap VPS w/ fast SSD storage" would have failed due to costs. Everybody wants Five Nines until they have to pay for it...

Re:DigitalOcean lost our server by nailer on Friday February 05, 2016 @01:18PM (#11043409)
Digital Ocean sell 'Simple Cloud Infrastructure'. Server data might be transient, but most web developers aren't infrastructure engineers and will have no idea of this - certainly it seems to have come as a surprise to some people. Digital Ocean need to manage their customer's expectations - and upsell for people who want permanent storage - as a rude shock doesn't do well for their brand.
Re:DigitalOcean lost our server by random_upvoter on Thursday February 04, 2016 @04:36PM (#11037445)
DO should do what's written in their terms of service, and the customer should read them carefully.
Re:DigitalOcean lost our server by nailer on Thursday February 04, 2016 @04:43PM (#11037487)
> DO should do what's written in their terms of service

Agreed.

> the customer should read them carefully.

Sure, but they shouldn't need to, as Digital Ocean should set expectations clearly.

Keep in mind Digital Ocean's <title>: Simple Cloud Infrastructure. Being surprised because there's an unsafe default hidden in a document somewhere didn't work out for MongoDB and it won't work out for DO.

Re:DigitalOcean lost our server by watty on Thursday February 04, 2016 @04:02PM (#11037199)
Isn't "non-issue" a big of an exaggeration? If the dry cleaners lost my clothes, the bank lost my money, a valet lost my car, or gmail lost my inbox I'd be angry.
Re:DigitalOcean lost our server by geofft on Thursday February 04, 2016 @04:19PM (#11037307)
If the bank lost the particular dollar bill that you deposited last week, but offered you a new dollar bill, would you get angry? If you rent a car from the airport every time you visit a city, and it's always been the same car, but you show up one night and they they tell you the car was in an accident and they'll get you another car, would you be angry?
Re:DigitalOcean lost our server by AznHisoka on Thursday February 04, 2016 @07:02PM (#11038297)
A more relevant question is: Would you be angry if you lost $500,000 because your bank only insured $150k?
Re:DigitalOcean lost our server by chc on Thursday February 04, 2016 @04:14PM (#11037272)
Yes, but those aren't expected outcomes when using those services. That's the difference gdgtfiend was pointing out — you should expect cloud servers to sometimes go away.
Re:DigitalOcean lost our server by bbcbasic on Thursday February 04, 2016 @04:23PM (#11037343)
They are expected outcomes. Banks fail (think 2008). Dry cleaners do lose clothes. Cars do get stolen. Plenty of stories in the news of all 3 of these. No so sure about gmail losing all your email, but I was able to accidentally access someone else gdocs once. I just logged in as me and saw a complete stranger's docs.
Re:DigitalOcean lost our server by nkozyra on Thursday February 04, 2016 @04:20PM (#11037322)
The expectation with these services you mentioned, however, is reliability.

The implicit (and explicit) volatility of cloud hosting should change that expectation with such services.

"Non-issue" is an exaggeration because it was potentially a catastrophe. But this is sort of the way these services "work." Servers/droplets/instances are ephemeral and replaceable, and their underlying data is not guaranteed in a failure.

Re:DigitalOcean lost our server by bobdole1234 on Thursday February 04, 2016 @09:19PM (#11038967)
The dry cleaner and bank don't tell you to make sure you have backups.