> And it looks like the attackers decided to start things back up again, as Linode.com is unavailable...
They're watching our status page for updates and starting new attacks when we resolve previous ones. There's been an almost 1:1 correlation lately.
This always rings hollow to me, and yet I hear it over and over.
A company like Linode surely has at least a dozen people who can be on call in a situation like this, probably much more. All it takes is for one engineer or even a product person... Heck a technically-minded support person could listen in on the war room meetings and get enough information to post something better than "we're fixing it".
5 minutes of blogging every six hours would be plenty.
And yet, people always claim it's impossible and there's no time. Frankly I find it frightening... If you are coding that fast that no one on your team has five minutes to step aside, take a breather after six hours of coding and summarize what the team just spent the last six hours doing, I shudder to think what kind of panicked alarmist interventions you are making.
There's no excuse for silence. There just isn't. It's a gross failure on the part of management to prioritize the responsibilities they have to your customers customers. Full stop. Sure, the engineers can't be expected to remember to tap out to blog. But if that's the extent of the accountability structures you can assemble during a crisis, that is a serious organizational failing, particularly for an organization the size of Linode.
Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey, we need to give some clarity to customers" are less available than the people whose time is spent firefighting.
We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.
I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires.
I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.
The timing of the DDoS was pretty interesting too, happening when not everyone is available.
Perhaps its time to consider some failover at another host. Same goes for anyone solely dependent on anyone.
Say I have one server setup on Linode London and one duplicated on DO Amsterdam. How do I quickly re-route the traffic from my two main load balancers London to Amsterdam without much propagation time?
* This is for an answer for the average "Hey I run a website or two" question. If you're asking and you are dealing with financial transactions or ad networks, well, you're asking in the wrong place and you should probably hire someone that knows how BGP anycasting works.
Eh, usually you just bgp anycast the DNS server. If you anycast just your DNS server, you are pretty much in the same pickle as if you host your dns with someone else who runs a reliable dns server (and have them do the anycast) - the pickle is that there's a minimum ttl.
Now you can set up a redirector server and anycast that, and like 3xx redirect all your requests (or more often just the first request) and that's better, really, 'cause that redirect can be changed quickly and on a per-request (or more often per-session) basis... but while I know a thousand companies that will sell you anycast DNS service, I don't know anyone who will sell someone with a vps customer budget an anycast'd redirector service.
Because they physically route the traffic, you can change backends pretty quickly and have traffic going to the new destination within seconds or minutes. Using DNS for end-users just has too much unpredictable latency, but it works well for routing the backend.
What happens with 1/10 second's database records not replicated? That could be a successful transaction on one, and no record of it on the other?
You can use a multi-master setup and strict consistency checks, though.
CloudFlare doesn't offer VPS instances. Google and Amazon are in a league of their own and, last I checked, don't offer the similar services at the same prices with the same level of support and same controls. Amazon might send you a nice bill at the end of the attack depending on what type it was.
DigitalOcean, as an example, routinely receives DDoS attacks and will just nullroute your VPS automatically until the attack ends.
When our customers are emailing and tweeting us and they just want to know when we are going to be up, and all we can say is "We have no idea, we don't know why this is happening or what's really going on", that's pretty much the definition of a worst case scenario from a customer service standpoint.
As someone whose business relies on Linode currently to function, I am sympathetic to Linode's plight...this is the equivalent of someone coming and setting off a bomb in your factory; not exactly something that you can always plan for even if you have prevention measures in place. But they would have kept a lot more of my sympathy long-term if they would have communicated better with their customers in the first place...
EDIT: And it looks like the attackers decided to start things back up again, as Linode.com is unavailable...