News for hackers. Stuff that matters.
Re:Linode turns 8; Disk space +25% by paraschopra on Thursday June 16, 2011 @02:51PM (#2662654)
Congrats on the anniversary. To provide a balanced perspective on this thread, let me share our experience. We (Visual Website Optimizer) are hosted on Linode since beginning but have experienced degradation of service lately particularly around network availability. There is at least one minor outage every month in their Newark data center (planning around network outage is much harder). While their support is responsive, these regular outages affect our business in a very significant manner. Every time we contact them, their staff just says these outages _happen_ and unavoidable and I cannot believe they are okay with it.

Maybe it is to do with their increased customer base, but earlier these outages were much less frequent and we enjoyed the service.

We own 25+ servers with them and have lately given thought of moving to some other provider. It's their responsive customer service and the pain of migrating everything that has kept us sticking to Linode.

Re:Linode turns 8; Disk space +25% by code_duck on Thursday June 16, 2011 @03:18PM (#2662786)
I notice a few blips now and then, too - it's visible on my graphs. Sometimes they can be as long as an hour or two. But then, I'm not getting any user reports in these times which makes me just wonder about the monitoring.

I was having some major problems with a server that would flip out under heavy load. Turns out it was the specific kernel I was using. Linode support was quite helpful once I actually contacted them about it.

Re:Linode turns 8; Disk space +25% by Dramatize on Thursday June 16, 2011 @04:38PM (#2663140)
I also get the blips which are visible on my graphs.
Re:Linode turns 8; Disk space +25% by callmeed on Thursday June 16, 2011 @03:03PM (#2662713)
If you have 25+ servers, aren't some of them in other data centers and configured in some sort of high-availability setup? What's your technology stack?

(FYI, I'm asking purely out of curiosity)

Re:Linode turns 8; Disk space +25% by paraschopra on Thursday June 16, 2011 @03:57PM (#2662967)
We have a backup set of servers at a different data center as well. But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center. A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue.

The only real solution to this is BGP routing which is complicated to setup and VERY expensive (only big guys do it). Here's some excellent answers on SO: http://serverfault.com/questions/153576/global-high-availabi...

Re:Linode turns 8; Disk space +25% by cbf on Friday June 17, 2011 @02:07AM (#2664530)
But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center.

All resolvers cache records for the TTL (or SOA minimum for negative lookups) specified by your authoritative servers.

A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue.

If the clients are web browsers they will try the alternative addresses in a record set if the one they pick doesn't respond.

Re:Linode turns 8; Disk space +25% by AgentConundrum on Friday June 17, 2011 @02:22AM (#2664566)
Forgive my ignorance, but isn't it normal to cache the DNS for 24 hours or so by default? I think it's typical (though I've never done it, so I'll apologize again if I'm talking out of my ass here) to drop the TTL down to basically nothing well ahead of the migration so that when you flip the switch, everyone will notice almost immediately, and then you can push the TTL back up to normal on the updated record.
Re:Linode turns 8; Disk space +25% by thaumaturgy on Thursday June 16, 2011 @02:58PM (#2662691)
Huh. I have a number of our pre-launch hosting customers on a web server in the Newark center; I get a text message within a few seconds of an outage, and haven't received any notifications in quite a while (unless it was due to my own error).

Maybe it's something on your particular rack or node? Have you tried setting up new Linodes in Newark and migrating to those?

Re:Linode turns 8; Disk space +25% by robflynn on Thursday June 16, 2011 @03:05PM (#2662724)
Which data center do you use? I have not experienced any outages with my linode server(s).
Re:Linode turns 8; Disk space +25% by nkassis on Thursday June 16, 2011 @03:27PM (#2662835)
How regular and how long? They don't provide 100% uptime guarantee. A few hours a month of downtime is what 99.99% uptime. That's not bad at all.

I personally haven't had issues with the network in the Dallas DC.

Re:Linode turns 8; Disk space +25% by rmah on Thursday June 16, 2011 @03:58PM (#2662972)
There are 30*24 = 720 hours a month. If a "few hours" is 2 hours, then that is 718/720 = 99.72% uptime. This means 0.28% downtime vs 0.01% downtime. IOW, almost 30x worse than a 99.99% uptime promise (if that's even what Linode promises).

Math FTW!

Re:Linode turns 8; Disk space +25% by nkassis on Thursday June 16, 2011 @04:39PM (#2663148)
ok.... seems like I can't enter numbers into irb ;p you are right 99.99% would be about an hour a year.
Re:Linode turns 8; Disk space +25% by paraschopra on Thursday June 16, 2011 @04:07PM (#2662997)
Network outage for us even for couple of seconds can be bad because we are serving about 1300 requests/second at peak which can mean a number of users getting affected.
Re:Linode turns 8; Disk space +25% by commx on Thursday June 16, 2011 @05:21PM (#2663330)
The Internet itself is inherently unreliable. Your users will traverse multiple hops (many networks) before connecting to your servers at any provider, as will any external monitoring systems you put in place. Brief issues along the paths between them and your servers can and will occur. While the Internet as a whole is designed to route around problems, this doesn't happen instantaneously in most cases, although it can happen very quickly.

This is not to say that internal network issues won't happen at any provider on occasion as well, because they certainly will. No network is perfect, least of all the massive system of interconnected networks known as the Internet, and you need to accept and plan for this. To believe or expect otherwise is simply unrealistic.

Re:Linode turns 8; Disk space +25% by steve___ on Saturday June 18, 2011 @12:28AM (#2667909)
We use the Newark data center and monitor our server with Nagios. In the past 90 days I have not received an alert because the connection was down.
Re:Linode turns 8; Disk space +25% by jsprinkles on Thursday June 16, 2011 @03:06PM (#2662726)
My Linode in Newark has not had network issues since last year. Acquaintance has a large deployment there with 100.0% availability. Positive this is not problems with your stack or monitoring?
Re:Linode turns 8; Disk space +25% by paraschopra on Thursday June 16, 2011 @04:04PM (#2662984)
No, it isn't. In addition to using Pingdom, we are continuously monitoring availability from multiple different locations (these scripts also do DNS switching). Every time we detect an issue, we contact Linode and many times they confirm network issues (sometimes internal, sometimes upstream).

Just two days back they confirmed one internal outage. See this screenshot: http://imgur.com/jYzuD

Almost every time we contact them about an outage, we get this drup response: "Apologize for inconvenience" which is okay, but they should at least upgrade their infra or take steps to avoid it.

Maybe your acquaintance is not monitoring the deployment with higher frequency. We monitor our stack multiple times every minute.

Re:Linode turns 8; Disk space +25% by hartror on Thursday June 16, 2011 @09:38PM (#2663986)
I am pretty happy with Linode for the most part and haven't experienced many issues. However when we have had issues, diagnosed the issue and filed a ticket we get always get told it is a known issue and "Apologize for inconvenience".

Why do I have to contact them to be told they are having an issue? Why can't they let me know asap so I can mitigate any problems they are having.

The last issue we had was their backup system was failing and by the time we discovered it, it had been broken for days.

Re:Linode turns 8; Disk space +25% by keidian on Thursday June 16, 2011 @10:56PM (#2664163)
You can check their status page at http://status.linode.com/ or drop by irc.oftc.net in the #linode channel. They also usually tweet as well.
Re:Linode turns 8; Disk space +25% by hartror on Monday June 20, 2011 @01:35AM (#2673011)
None of which they had done. And their backup system isn't exactly something straight forward to setup monitoring of to notice there was an issue.
Re:Linode turns 8; Disk space +25% by jsprinkles on Thursday June 16, 2011 @04:20PM (#2663056)
Pingdom is notoriously unreliable and I base this on personal experience and experiences of others. False positives are the norm and it is garbage.
Re:Linode turns 8; Disk space +25% by paraschopra on Thursday June 16, 2011 @04:50PM (#2663200)
Maybe, but we do monitor using our scripts as well.