News for hackers. Stuff that matters.

Digital Ocean NYC2 Droplets Down

And counting. Second time in about a week and during prime time hours. If it doesn't get up quickly or if happens again to NYC2 in the next few weeks may be time to reconsider DO for hosting production sites. I love them, and glitches happen, but twice in a row clients have a hard time swallowing.
( 68 comments | 52 points | on HN )
Re:Digital Ocean NYC2 Droplets Down by hadoukenio on Tuesday July 29, 2014 @05:12PM (#8105051)
Why do people have their eggs in one basket? More specifically, whenever hosting comes up on HN, it's always DO vs Linode. Why not Zoidberg^Wboth?

In the age of DevOps, it's not hard to configure different providers. I have been using Linode and DO for a while now. I noticed DO go down last week, checked my site, and it was UP because it was being served via Linode.

I flinch every time I hear someone say to me that their service has redundancy because they use multiple Amazon regions. Multiple providers people! And preferably multiple providers in multiple states, paid for by mutually exclusive credit cards!

Redundancy doesn't mean having multiple hosts by the same provider, within the same data center, within the same rack, provided by the same power strip.

tl;dr: Plan for outages.

Re:Digital Ocean NYC2 Droplets Down by opendais on Tuesday July 29, 2014 @06:00PM (#8105277)
It is always DO vs. Linode because both are relatively stable, relatively good for their price points. Tbh, the only real reason I use Linode over DO is the fact I've seen higher restarts on my DO nodes and the fact I've attracted DDoS's at a size Linode shrugs about.
Re:Digital Ocean NYC2 Droplets Down by anderspetersson on Tuesday July 29, 2014 @06:15PM (#8105335)
How do you do this on the DNS layer? Very short TTL and switching on failure detection? Do you use both providers at the same time when both are up via DNS round robin or similar?
Re:Digital Ocean NYC2 Droplets Down by hadoukenio on Tuesday July 29, 2014 @06:29PM (#8105385)
There's nothing stopping you from adding multiple A records for a domain. In fact, that's what a lot of people do:

  dig a www.google.com

  www.google.com.         300     IN      A       74.125.237.178
  www.google.com.         300     IN      A       74.125.237.177
  www.google.com.         300     IN      A       74.125.237.176
  www.google.com.         300     IN      A       74.125.237.179
  www.google.com.         300     IN      A       74.125.237.180
Browsers these days will resolve and try each one, on a first-come-first-served basis. If one fails, it will try the request on the next server.
Re:Digital Ocean NYC2 Droplets Down by stevekemp on Wednesday July 30, 2014 @08:15AM (#8107621)
If you have your own IP-space you can use BGP to route IPs to different locations.

If you're relying on DNS then short TTLs are pretty reliable, but there can be extended outages due to erroneous client-caching.

Re:Digital Ocean NYC2 Droplets Down by bigredtech on Tuesday July 29, 2014 @02:44PM (#8104110)
I'm a big fan of Linode. Have had one of their mid-range servers for about 2 years now and have had no troubles. When it came time to upgrading, or any other questions I had, their support staff is 1) quick, 2) knowledgable, 3) thorough & helpful.

I've used DO in the past for quick spin-ups, and they're great for that. But for long running applications I'd stay away. Was actually considering DO for a new project, but this kind of tipped the scales the other way.

If you're looking to switch, I think Linode gives you the best bang and reliability for your buck. Have also used Rackspace for a previous startup. A bit more pricey, but also a great service if you're considering switching.

Re:Digital Ocean NYC2 Droplets Down by meritt on Tuesday July 29, 2014 @02:59PM (#8104228)
Linode can't silo resources on a per VPS basis and constantly will harass you for over-utilizing your share of arbitrarily enforced limits. Just a heads up if you actually utilize your instances.
Re:Digital Ocean NYC2 Droplets Down by harpastum on Tuesday July 29, 2014 @03:05PM (#8104270)
Are you talking about the "Linode Alert" emails? Those are user-configurable — in the email, they specifically state "This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds based on your own individual needs."

If that's not what you're talking about, I apologize, but I've been using Linode for years (often with very compute-intensive projects) and I haven't seen what you're talking about.

Re:Digital Ocean NYC2 Droplets Down by jarin on Tuesday July 29, 2014 @03:14PM (#8104339)
Just out of pure laziness, I use the CPU and disk access emails as makeshift notifications for when my backups run.
Re:Digital Ocean NYC2 Droplets Down by bigredtech on Tuesday July 29, 2014 @03:08PM (#8104287)
Have run a web-app with 30K+ users. Only been 'harassed' when I've been warned that I'm running the CPU at 350% of max rate and I should be concerned. That's ok in my book.
Re:Digital Ocean NYC2 Droplets Down by Ronsenshi on Tuesday July 29, 2014 @03:37PM (#8104483)
I find that surprising. Linode support never contacted me regarding resource usage. Not even when one of the servers has been under DDOS. I contacted them first notifying of such event. And that was it.

I agree with harpastum. I think you're confusing automatic email notifications based on your own settings with Linode support. I feel like you have to seriously abuse their infrastructure to get them to contact you.

Disclaimer: has been a happy Linode customer for over 4 years now.

Re:Digital Ocean NYC2 Droplets Down by meritt on Tuesday July 29, 2014 @03:48PM (#8104544)
http://i.imgur.com/UCPWdBq.png

I was not confusing automated emails with direct contact from Linode support. I fully understand and can appreciate the need to be a "nice neighbor" in a shared environment but what baffles me is:

1) Linode doesn't have the capability to actually enforce any sort of quota or limit. It's 2014, there's been numerous advancements in the area of virtualized servers in the past decade. cgroups is a very easy way that works across xen, kvm and lxc.

2) Linode doesn't publish these arbitrary limits nor give guidance on which node size comes with what acceptable usage levels.

Re:Digital Ocean NYC2 Droplets Down by guiambros on Tuesday July 29, 2014 @04:56PM (#8104966)
This is ridiculous. Would suggest you move out of Linode right away.

It's their job to limit resources on your node, not your job to self-limit. Asking you to limit resources because it's "affecting other customers' performance" is not only outrageous, but also unfair with the other customers being impacted.

This reminds me of the late 90's [1], when web hosting meant sharing the same box with hundreds of other customers, and very limited tools to enforce limits. When you had a resource hog, the only option was really to get them to reduce usage, or move them to a separate box (which wasn't as easy as today).

[1] ps: I know this well; used to have a web hosting company at the time. We had to frequently move customers around, and in some cases to terminate their contracts - e.g., porn sites. But this was a good 10 years before Xen become commonplace.

Re:Digital Ocean NYC2 Droplets Down by meritt on Tuesday July 29, 2014 @05:09PM (#8105033)
Yup. That's the conclusion we reached and also why I suggested approaching Linode with caution. They're attempting to compete with the "cloud" providers but running their business the same as they did a decade ago.
Re:Digital Ocean NYC2 Droplets Down by Ronsenshi on Wednesday July 30, 2014 @01:48AM (#8106609)
Thanks for the answer.

1) I don't know enough to talk about resource limiting that Linode might do. There might be a reason they don't want to or can't use it with their infrastructure.

2) I agree, they could publish some guidelines for resource usage, but this is hardly baffling. I'd guess most of the customers just don't hit this problem, so there's no need to list all of the limitations.

Re:Digital Ocean NYC2 Droplets Down by opendais on Tuesday July 29, 2014 @06:05PM (#8105301)
Thanks for explaining. :)

Anything I have that uses that level of disk i/o I don't run in a VPS which would explain why I've never noticed before.

Re:Digital Ocean NYC2 Droplets Down by opendais on Tuesday July 29, 2014 @03:05PM (#8104273)
Really? I find that odd. I do load testing on their instances all the time. which maxes out cpu/disk to give me an idea of performance changes for stuff I have high throughput on.

Hell, I've gotten DDoS'd for like 2 Gbps and they were like "Oh, ya, its fine."

So, can you explain more about what happened?

Re:Digital Ocean NYC2 Droplets Down by nilved on Tuesday July 29, 2014 @03:08PM (#8104288)
If you'd used their servers for a bit longer, you'd have plenty of trouble. Linode is notorious for dreadful security practices. Their main ColdFusion app (yes, really) has been hacked on numerous occasions, including at least once incident that involved customer data loss. Linode dodged the issue and didn't tell their customers until badgered.

I can agree that DO isn't suited for production apps, but Linode isn't either. They're both VPS playgrounds.

Re:Digital Ocean NYC2 Droplets Down by antsar on Tuesday July 29, 2014 @03:26PM (#8104411)
Is there a better alternative you would suggest? While Linode has done better than DO, that fiasco left me feeling somewhat uneasy as well.
Re:Digital Ocean NYC2 Droplets Down by nilved on Saturday August 02, 2014 @07:03PM (#8126356)
There are a great many VPS and server providers out there. I've had good experiences with gandi.net, Rackspace, moln.is and pgrmr. I've seen Ramnode and Lithium Hosting get a lot of praise. You could also use AWS if you don't have any ethical objections to using Amazon products.
Re:Digital Ocean NYC2 Droplets Down by Ronsenshi on Tuesday July 29, 2014 @03:40PM (#8104504)
Same question as antsar. What would you suggest then?
Re:Digital Ocean NYC2 Droplets Down by olso on Tuesday July 29, 2014 @03:22PM (#8104385)
I use DO because they have servers in Amsterdam, which happens to be very close to my country.

I never had problems with DO that weren't resolved promptly.

Just a note, Linode is very US-centric anyway... (and that one London location doesn't make it international very much)

Re:Digital Ocean NYC2 Droplets Down by kl4m on Tuesday July 29, 2014 @02:41PM (#8104082)
Their status page indicates "We have identified a slow connection to our NYC2 data center". Assuming they have redundant connections, this seems like a big understatement for being down for 1 hour and going. Even their control panel is down for me. ( http://i.imgur.com/3ltg00o.png )
Re:Digital Ocean NYC2 Droplets Down by neom on Tuesday July 29, 2014 @02:48PM (#8104149)
we've had an ongoing issue with a juniper fiber core in nyc2 it seems. We've engaged juniper to help us, I don't expect this to be an issue again.
Re:Digital Ocean NYC2 Droplets Down by jcampbell1 on Tuesday July 29, 2014 @02:34PM (#8104021)
Linode's new offering probably the better choice these days. I get the feeling that DO is just overwhelmed with problems related to their hyper growth - 10% per month on a huge base.

http://trends.netcraft.com/www.digitalocean.com

Re:Digital Ocean NYC2 Droplets Down by mctDarren on Tuesday July 29, 2014 @02:38PM (#8104059)
They've done so well to this point keeping up with the growth curve. Hope they don't become crippled by success.
Re:Digital Ocean NYC2 Droplets Down by jqueryin on Tuesday July 29, 2014 @02:32PM (#8104007)
Having previously left Linode for Digital Ocean due to lower pricing and SSDs, I can safely say Linode both dropped their prices and matched the DO plans with SSDs included. If you want to jump ship, that's the route I'd go for a comparable.

I still love DO, but downtime is a no-no when it comes to VPSes.

Re:Digital Ocean NYC2 Droplets Down by JohnTHaller on Tuesday July 29, 2014 @02:40PM (#8104078)
When you charge so little and the SLA is just credit during the time of the outtage (pennies), downtime is pretty much a given. It always surprises me that this surprises people.
Re:Digital Ocean NYC2 Droplets Down by ramatgan on Tuesday July 29, 2014 @02:34PM (#8104022)
In the process of switching over now.
Re:Digital Ocean NYC2 Droplets Down by JohnTHaller on Tuesday July 29, 2014 @02:47PM (#8104133)
You should probably reconsider hosting production sites on super-cheap VPSes with a 1:1 SLA refund. They're going to go down. And, when they do, you'll get the 7/10ths of a cent refund per hour promised for your $5 instance (1.4 cents per hour for $10, 2.8 cents per hour for $20, etc). But if you depend on a site for your business, ask yourself how much it costs you per hour of downtime and plan appropriately.

For context: I'm a happy Digital Ocean customer (as well as Rackspace and others). I like their services and consider them a great value for the money. I've also gotten multiple developer friends into using them. But I don't let them be a single point of failure for anything critical.

Re:Digital Ocean NYC2 Droplets Down by kbar13 on Tuesday July 29, 2014 @02:49PM (#8104157)
downtime is an issue no matter how much you pay your infrastructure provider. You should account for downtime in your application and your infra by taking advantage of low-hanging high availability fruit.

Things like saltstack (saltstack.org) make this mega-easy.

Re:Digital Ocean NYC2 Droplets Down by JohnTHaller on Tuesday July 29, 2014 @02:57PM (#8104214)
It's easier for apps since you can build in failover support based on multiple servers pretty easily (I just implemented this in our 'live' installers recently on Windows). It's a bit harder at the website level for an average user, though. I know far too many people that have a single point of failure website on a super-cheap VPS that's just down whenever the VPS is down. Most of them would have no idea how to implement a load balancer and failover support. Though I could probably get them into something like CloudFare or similar.
Re:Digital Ocean NYC2 Droplets Down by Sir_Cmpwn on Tuesday July 29, 2014 @02:46PM (#8104125)
I am getting pretty sick of it, too. I might go back to Linode. I was already upset that they dropped Arch Linux support.
Re:Digital Ocean NYC2 Droplets Down by opendais on Tuesday July 29, 2014 @03:07PM (#8104286)
Linode has price matched them so its not a price issue anymore. [e.g. $10 Linode is roughly the same as $10 DO]

Glad to see the ProjectWonderful ads providing something, even if its small btw. :)

Re:Digital Ocean NYC2 Droplets Down by Sir_Cmpwn on Tuesday July 29, 2014 @03:26PM (#8104418)
It definitely helps. The biggest source of income has been (surprisingly) Gittip, though.
Re:Digital Ocean NYC2 Droplets Down by opendais on Tuesday July 29, 2014 @03:59PM (#8104611)
Ya, I am surprised by that too. I'd create one but I'd feel guilty. My ads cover my costs.
Re:Digital Ocean NYC2 Droplets Down by wglb on Tuesday July 29, 2014 @03:52PM (#8104575)
Er, no.

My droplets are up. Never went down today.

Also, please don't post that things are down here on HN. This is not a support forum. Not for DO, not for AWS.

Re:Digital Ocean NYC2 Droplets Down by iurisilvio on Tuesday July 29, 2014 @04:02PM (#8104634)
You came here after it was fixed. It is fine now. Also, it was just NYC2, maybe only part of it.
Re:Digital Ocean NYC2 Droplets Down by wglb on Tuesday July 29, 2014 @04:19PM (#8104748)
Mine are on NYC2. Terminal sessions up all day. On three of them.

Check http://downforeveryoneorjustme.com instead of HN.

Re:Digital Ocean NYC2 Droplets Down by anatoli on Tuesday July 29, 2014 @04:33PM (#8104845)
Or you could check the official status page[1] and see that there was indeed an outage.

[1] https://status.digitalocean.com/

Re:Digital Ocean NYC2 Droplets Down by wglb on Tuesday July 29, 2014 @04:36PM (#8104869)
Even further reason not to post it here!
Re:Digital Ocean NYC2 Droplets Down by uptown on Tuesday July 29, 2014 @02:33PM (#8104015)
Here's their status panel: https://status.digitalocean.com/
Re:Digital Ocean NYC2 Droplets Down by potomushto on Tuesday July 29, 2014 @02:38PM (#8104057)
https://cloud.digitalocean.com/ now is unavailable too.
Re:Digital Ocean NYC2 Droplets Down by larrys on Tuesday July 29, 2014 @03:02PM (#8104248)
"but twice in a row clients have a hard time swallowing."

May I ask what your clients are paying?

Maybe the answer is to charge them more instead of giving them a low price (an assumption on my part of course which is why I ask what you are getting for this) and some implicit guarantee of uptime.

For example we had some customers some time ago who were pooled on a shared IP address for sending SMTP service. Occasionally they would experience problems with mail reputation on that shared IP. If they complained about that would simply explain that that was the drawback of a shared IP and that they could always upgrade to a service level where they got a unique IP and could establish their own reputation.

Re:Digital Ocean NYC2 Droplets Down by opendais on Tuesday July 29, 2014 @03:10PM (#8104307)
Folks, I strongly, strongly suggest if you use Cloud/VPS providers to look into something like Cassandra, Galera, Mongo, etc. that can do WAN replication and using 3 DCs.

Yes it costs money but things like this are avoidable. Every single Datacenter on the planet has downtime and/or performance issues from time to time. Hell, one of the places I have a dedi at had its network performance drop by 50% for about 15 minutes while they did emergency maintenance.

Things like this are going to happen as long as you depend on one provider.

Re:Digital Ocean NYC2 Droplets Down by Donzo on Tuesday July 29, 2014 @03:05PM (#8104271)
I've had great experiences with them. I recommended DO to everyone.

Now I'm tempted to drive to New York and punch them each in the face.

PEAK downtime.

EDIT: $160 refund. Maybe just a Charlie horse.

Re:Digital Ocean NYC2 Droplets Down by publicfig on Tuesday July 29, 2014 @02:27PM (#8103960)
I just created a droplet today, and was trying for the life of me to figure out what I would have done so early on in setting it up to cause it to become unresponsive. Glad to see it's not just me. It's not really a great sign though, as I was hoping to use this fairly frequently. Is this a common occurrence, or do I just have bad luck? If so, I'll definitely have to find another solution.
Re:Digital Ocean NYC2 Droplets Down by zhodge on Tuesday July 29, 2014 @02:33PM (#8104010)
I'm not too qualified to comment on the hosting market, but I do want to vouch for DigitalOcean as I've never had uptime issues with them until now. They also have always had timely and helpful support and their pricing is obviously competitive.

That being said, near 100% uptime is really desired in this space and I've been scratching my head for the past hour or so wondering why I couldn't connect to my mongo process running on my droplet...maybe I should have tested DO sooner, but the fact that only mongo seemed to be failing (and not postgres for example) made me think I had client and not server issues.

Re:Digital Ocean NYC2 Droplets Down by gingerlime on Tuesday July 29, 2014 @02:40PM (#8104073)
I've been using Linode for many years, and DO for some time now (a year? not sure, since I can't even login to their control panel right now and check). I've seen issues on both providers, and dealt with support of both not too infrequently.

I'd still say Linode is miles ahead in terms of expertise and responsiveness. Most of my servers are on Linode for this reason, and only a few on DO.

As a customer I enjoy this competition and would like to keep both players compete - because I'm benefiting from it. So I'm likely to stick to both. But I would still recommend Linode more easily than I would DO.

Re:Digital Ocean NYC2 Droplets Down by zhodge on Tuesday July 29, 2014 @02:48PM (#8104150)
Thanks for your input. I might be switching over as a result of the recent downtime and the support Linode has been receiving on this thread. Unfortunately, especially for non-hobbyist apps and services, this is an area demanding fickleness.
Re:Digital Ocean NYC2 Droplets Down by heavymark on Tuesday July 29, 2014 @02:42PM (#8104100)
The big issue here is NYC2 droplets went down only 7 days ago. This time its even longer at over an a half. When it happened last week it was unfortunate, but things happen. When it happens twice in a week that is a much bigger problem.
Re:Digital Ocean NYC2 Droplets Down by zhodge on Tuesday July 29, 2014 @02:45PM (#8104123)
Agreed. I think what worries me most is that I wasn't aware of their downtime a week ago..
Re:Digital Ocean NYC2 Droplets Down by lemonberry on Tuesday July 29, 2014 @03:57PM (#8104599)
They sent out an email last week with an explanation.
Re:Digital Ocean NYC2 Droplets Down by mbreedlove on Tuesday July 29, 2014 @02:24PM (#8103935)
I'd love to know what's going on. My website hosted on DO, michaelbreedlove.net, is unavailable. I also have a few services running on subdomains of the same DO VPS instance; they're all accessible. I also have SSH access. They seem to be having some serious HTTP routing issues.
Re:Digital Ocean NYC2 Droplets Down by alokyadav15 on Tuesday July 29, 2014 @02:41PM (#8104081)
Re:Digital Ocean NYC2 Droplets Down by marcusr on Tuesday July 29, 2014 @02:09PM (#8103819)
It seems to be http (perhaps load balancers) rather than the instances being down - I'm ssh-ed into a droplet and everything is working fine, except remote HTTP connections.
Re:Digital Ocean NYC2 Droplets Down by toomuchtodo on Tuesday July 29, 2014 @02:32PM (#8104006)
What happens when you try to execute outbound web requests from the instance?

http://netalyzr.icsi.berkeley.edu/cli.html

Re:Digital Ocean NYC2 Droplets Down by iurisilvio on Tuesday July 29, 2014 @02:15PM (#8103874)
I ssh-ed into some droplets, but others are completely inacessible outside DO. The internal network works fine, so it is just a several network outage. Again.
Re:Digital Ocean NYC2 Droplets Down by JohnTHaller on Tuesday July 29, 2014 @02:42PM (#8104088)
I've been unable to SSH in for an hour and a half. But I expected this when I signed up (cheap = cheap) so have backups running that were switched to.
Re:Digital Ocean NYC2 Droplets Down by publicfig on Tuesday July 29, 2014 @02:31PM (#8103993)
I'm struggling to ssh into droplets, but it seemed like when I was ssh'd in, I didn't have an issue at all.
Re:Digital Ocean NYC2 Droplets Down by mikeash on Tuesday July 29, 2014 @02:40PM (#8104076)
It seems to be intermittently up and down. So your connections all die, and you can't reconnect for a while, then it comes back and everything looks good for a bit until it dies again.
Re:Digital Ocean NYC2 Droplets Down by heavymark on Tuesday July 29, 2014 @02:46PM (#8104129)
Same here. Since 1:24p, Pingdom has reported our sites going up and down about 16 times.
Re:Digital Ocean NYC2 Droplets Down by rrss1122 on Tuesday July 29, 2014 @03:50PM (#8104560)
We're getting the same thing. I don't care how cheap it is or how nice the customer service is, but if they are not up 99% of the time they will not be my web host.
Re:Digital Ocean NYC2 Droplets Down by JohnTHaller on Tuesday July 29, 2014 @02:59PM (#8104231)
I'm back up as of 2:59pm EDT, so I was down for about 1:35. This is the only outtage longer than a few minutes I've had with DO thusfar.
Re:Digital Ocean NYC2 Droplets Down by dangoldin on Tuesday July 29, 2014 @02:34PM (#8104020)
One of my instances on NY2 is fine and the other one is sporadically accessible via SSH. Hope they figure this one out soon.
Re:Digital Ocean NYC2 Droplets Down by bmoresbest55 on Tuesday July 29, 2014 @02:36PM (#8104039)
This is a little ridiculous. I hope not to have this problem again. Second strike her. Unfortunately.
Re:Digital Ocean NYC2 Droplets Down by alokyadav15 on Tuesday July 29, 2014 @02:57PM (#8104218)
its working now , atleast for me .
Re:Digital Ocean NYC2 Droplets Down by n0nenet on Tuesday July 29, 2014 @02:04PM (#8103775)
I have to agree with you on this one.