News for hackers. Stuff that matters.
Re:Linode is suffering on-going DDoS attacks by thrownaway2424 on Tuesday December 29, 2015 @10:50AM (#10807197)
I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.
Re:Linode is suffering on-going DDoS attacks by scurvy on Tuesday December 29, 2015 @11:45AM (#10807507)
What DDoS protection does AWS provide? The only thing mentions on their webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection strategy is to open up your wallet.

About 6 months ago they did hire Jeff from BlackLotus. Given that timeline, I'd expect them to announce some sort of DDoS protection offering in the next few quarters.

Edit to be more specific: AWS gets hit with a lot of DDoS attacks. While all of AWS isn't unreachable during an attack, parts of it are. It's so large that you might not notice, but parts are unreachable. AWS/GCE size only makes it less noticeable, but they have no customer facing DDoS protection offerings. Their only offering is to buy more of their services. These providers don't have magical 1000000gbps links. They're regular 100gbps links (or 100gbps LACP channels) that can get overrun in large enough attacks.

Re:Linode is suffering on-going DDoS attacks by ivan_ah on Wednesday December 30, 2015 @09:48AM (#10812225)
> What DDoS protection does AWS provide? >

I've got no internal knowledge of how AWS operates, but I once went too far when load testing a new AWS machine (in terms of reqs/sec), and the traffic started getting dropped somewhere before reaching the box. Not sure if it was an elastic ip or behind an ELB, but I found it interesting.

Re:Linode is suffering on-going DDoS attacks by justinholmes on Tuesday December 29, 2015 @02:17PM (#10808451)
Actually "Amazon CloudFront also has filtering capabilities to ensure that only valid TCP connections and HTTP requests are made while dropping invalid requests. This takes the burden of handling invalid traffic (commonly used in UDP floods, SYN floods, and slow reads) off your origin."

and

"By using multiple PoPs, Amazon CloudFront has the inherent ability to help mitigate against both infrastructure and some application layer DDoS attacks by dispersing the traffic across multiple locations."

source: https://d0.awsstatic.com/whitepapers/DDoS_White_Paper_June20...

Re:Linode is suffering on-going DDoS attacks by nacs on Tuesday December 29, 2015 @04:05PM (#10809059)
Amazon Cloudfront is a CDN. A significant portion of a CDNs job is to be highly redundant and to handle DDoS.

If you're comparing Amazon's offering to Linode, you should really compare to the protection Amazon offers on EC2 and similar VPS-like products.

Re:Linode is suffering on-going DDoS attacks by justinholmes on Tuesday December 29, 2015 @04:46PM (#10809292)
I was merely answering the question "What DDoS protection does AWS provide?".

Normally you wouldn't expose a EC2 instance without putting cloudfront, elbs in front of it and locking down incoming traffic to cloudfront IPs.

Re:Linode is suffering on-going DDoS attacks by gist on Tuesday December 29, 2015 @12:08PM (#10807618)
"I'd expect them to announce some sort of DDoS protection offering in the next few quarters."

It will be very interesting to see how cost effective that protection actually is and what they charge for it. Nice profit center off of FUD.

Re:Linode is suffering on-going DDoS attacks by scurvy on Tuesday December 29, 2015 @12:23PM (#10807729)
If they price it like cheap insurance, they will make a lot of money and offer a worthwhile service. Pay them a low monthly rate (X) to have the service active and then pay hourly at Y rate during an attack. You can play with X and Y to maximize your monthly revenue while still offering a cheap (but very valuable) service that retains your customers.
Re:Linode is suffering on-going DDoS attacks by thrownaway2424 on Tuesday December 29, 2015 @12:41PM (#10807862)
I don't know, but all traffic to GCE is routed through Google's frontend, which provides in-built DDoS protections.
Re:Linode is suffering on-going DDoS attacks by scurvy on Tuesday December 29, 2015 @12:49PM (#10807921)
I'd imagine they use VRF's to quickly segment the traffic after ingress. Google.com might have DDoS protection, but I'm wary that it extends to GCE. I've read about Google Andromeda, but there's no real meat in any article about DDoS mitigation.
Re:Linode is suffering on-going DDoS attacks by thrownaway2424 on Tuesday December 29, 2015 @01:21PM (#10808139)
This document specifically claims that "All traffic is routed through custom GFE (Google Front End) servers to detect and stop malicious requests and Distributed Denial of Service (DDoS) attacks."

https://cloud.google.com/security/whitepaper

Re:Linode is suffering on-going DDoS attacks by scurvy on Wednesday December 30, 2015 @07:47PM (#10815445)
How though? There's remarkably little information in there for being a whitepaper. If all they do is drop Christmas-tree packets and bogus UDP/ICMP traffic, that's not much in the way of protection. I'd like to see exactly what/how they're doing. How do they know what traffic is malicious? Do you get a control panel to block certain L7 traffic? What L7 inspectors do they support?

Sorry, but I'm not buying it.

Re:Linode is suffering on-going DDoS attacks by skj on Tuesday December 29, 2015 @12:54PM (#10807969)
It does extend to GCE, to an extent. The reason for this is that if someone is DDOSing your GCE VMs, they're going to be affecting the network performance of unrelated projects. So, Google has to provide some protection so that, at a minimum, other customers are not affected.
Re:Linode is suffering on-going DDoS attacks by workitout on Tuesday December 29, 2015 @07:12PM (#10809963)
So...Google can stop all DDOS's? Everyone should dump AWS, Linode, Cloudflare, etc. and go to Google for their VMs then.
Re:Linode is suffering on-going DDoS attacks by mwcampbell on Tuesday December 29, 2015 @11:10AM (#10807310)
What about DigitalOcean? Its pricing is comparable to Linode's, yet DigitalOcean is now the second largest web host in the world according to Netcraft. Do you still think that AWS, Google Cloud Platform, and Microsoft Azure offer better protection?
Re:Linode is suffering on-going DDoS attacks by ivank on Tuesday December 29, 2015 @11:47AM (#10807512)
Here's the email I received (many times) when someone sent a smallish 1Gbit/s DDoS to my digitalocean server:

> Our system has automatically detected an inbound DDoS against your droplet named xyz with the following IP Address: xx.xx.xx.xx

> As a precautionary measure, we have temporarily disabled network traffic to your droplet to protect our network and other customers. Once the attack subsides, networking will be automatically reestablished to your droplet. The networking restriction is in place for three hours and then removed.

> Please note that we take this measure only as a last resort when other filtering, routing, and network configuration changes have not been effective in routing around the DDoS attack.

> Please let us know if there are any questions, we're happy to help.

This happened mere seconds after the DDoS begun! Therefore they lied about having tried to mitigate the attack.

No amount of contacting support got me un-blackholed before the 3 hour mark, and when I popped back into the network, I was blackholed again for another 3 hours...

I moved to a $3.50/mo OpenVZ VPS at OVH, and OVH's VAC system soaked up the DDoS just fine.

Re:Linode is suffering on-going DDoS attacks by nacs on Tuesday December 29, 2015 @01:04PM (#10808031)
I've received the same type of email multiple times from DigitalOcean and its extremely frustrating.

With even the smallest of traffic spikes, DigitalOcean will detect it as a DDoS and immediately cut off your server for 3 hours.

If even a typical (< 100Mbit) broadband-cable connection hits your server with a spike of traffic for less than 5 minutes, your server will be taken offline for 3 hours minimum.

I've used multiple VPS providers and dedicated-server providers and DO is absolutely the worst when it comes to DDoS policy.

Re:Linode is suffering on-going DDoS attacks by solusipse on Tuesday December 29, 2015 @02:40PM (#10808594)
That's why DigitalOcean isn't ready for production (unless hiding ip address behind, lets say, cloudflare is an option in your case) and that's why I decided not to use their services anymore.
Re:Linode is suffering on-going DDoS attacks by mwcampbell on Tuesday December 29, 2015 @03:52PM (#10808994)
Just curious, what do you use instead?
Re:Linode is suffering on-going DDoS attacks by cddotdotslash on Tuesday December 29, 2015 @12:57PM (#10807990)
If you even have the slightest inkling that your site will be attacked by a DDOS (either targeted or randomly) please reconsider DigitalOcean. Their method of handling a DDOS is basically "your node is being attacked, so we're disconnecting it from the network for a few hours. Try to figure out why someone is attacking you." Then, the machine gets reconnected three hours later and the attack cycle continues. Not to mention that since the machine is offline, there is no way of logging in to check access logs, modify firewalls, etc. I've never dealt with a company like DO before. Switched to AWS last year and, even when I was getting extreme traffic, AWS never just shut down access without notice.
Re:Linode is suffering on-going DDoS attacks by deno on Tuesday December 29, 2015 @11:23AM (#10807392)
OVH offers very comprehensive DDoS protection with all but the most budget servers:

https://www.ovh.com/us/anti-ddos/

Re:Linode is suffering on-going DDoS attacks by MichaelGG on Tuesday December 29, 2015 @11:26AM (#10807414)
Comments from people seem to indicate their anti DDoS was "we blackhole you if you draw an attack". Has that changed?
Re:Linode is suffering on-going DDoS attacks by deno on Tuesday December 29, 2015 @11:35AM (#10807453)
Yes. This hasn’t been true for a long time. OVH runs a huge network and they invested in tools to mitigate and neutralize DDoS attacks[1]. Of course an attack can still saturate your servers’ NICs but they won’t drop you anymore.

At their scale they are basically forced to handle big attacks on regular basis. The fact that they offer this protection in their basic package is what makes them a great host[2].

They don’t get a fraction of the love Cloudflare gets, but they should.

[1] https://www.ovh.com/us/anti-ddos/hoovering-up.xml

[2] https://www.ovh.com/us/news/articles/a1171.protection-anti-d...

Re:Linode is suffering on-going DDoS attacks by tombrossman on Tuesday December 29, 2015 @01:12PM (#10808088)
Good to see OVH getting some recognition here. I am a happy customer of both OVH and DigitalOcean but my usage has become strongly weighted towards OVH. The OVH web interface could be improved (I've had to open support tickets for basic things like activating auto-renewal), that's the only downside with them really. Range of products, network, disk speed and responsiveness are all consistently very good. Their new 2016 'Cloud' VPSs have markedly better disk speed than DigitalOcean's SSDs, which are already pretty fast.

For a quick side project I still like DigitalOcean's hourly billing and user interface, but any machine I plan on using for a month or more are all with OVH by default.

Re:Linode is suffering on-going DDoS attacks by X-Istence on Tuesday December 29, 2015 @12:41PM (#10807868)
Their Anti-DDoS setup is amazing. Once they detect an attack it goes through their VAC setup, and legitimate traffic will still reach you, but illegitimate traffic won't.

Got hit with a DDoS to my DNS servers, I didn't see any external service disruption thanks to OVH's setup. I have several of their cheaper SoYouStart servers, and it's awesome.

Re:Linode is suffering on-going DDoS attacks by pizzeys on Thursday December 31, 2015 @11:51AM (#10818372)
Yes. We moved an IRC server that was under attack (and being dropped from DO/Linode) there, and it ate it fantastically well for the price. I've never seen anyone else do it that well for that cheap, to be honest.
Re:Linode is suffering on-going DDoS attacks by sumedh on Thursday December 31, 2015 @07:47PM (#10820603)
Why is ovh so cheap compared to linode and DO, whats the catch?
Re:Linode is suffering on-going DDoS attacks by jedicoffee on Tuesday December 29, 2015 @01:42PM (#10808277)
Yea, throw money at Netcraft and you can be batman of the internet. The numbers being provided to the public are not accurate in any way.
Re:Linode is suffering on-going DDoS attacks by Mahn on Tuesday December 29, 2015 @11:16AM (#10807350)
As a Linode customer, I guess this wouldn't have happened with AWS, but the way they handle it is still far better than "your server, your bandwidth, deal with it" that you'd get with e.g. Hetzner dedicated servers. We had several hours of downtime, which sucks, but other than that we didn't have to worry about taking measures against the DDoS ourselves.
Re:Linode is suffering on-going DDoS attacks by deno on Tuesday December 29, 2015 @11:22AM (#10807383)
On AWS the DDoS hits the wallet instead. What’s the difference?

It’s “denial of service,” not denial of server and network resources.

Re:Linode is suffering on-going DDoS attacks by otterley on Tuesday December 29, 2015 @01:11PM (#10808084)
AWS ELBs don't forward DNS and NTP traffic to the backends, and ingress bandwidth isn't billed to you. So there's no impact on the wallet when the most common DDoS traffic is aimed at you.
Re:Linode is suffering on-going DDoS attacks by deno on Tuesday December 29, 2015 @01:34PM (#10808210)
That’s a great point. Still leaves you open for a lot of abuse though.
Re:Linode is suffering on-going DDoS attacks by otterley on Tuesday December 29, 2015 @01:43PM (#10808281)
What would be an effective alternative?
Re:Linode is suffering on-going DDoS attacks by deno on Tuesday December 29, 2015 @02:04PM (#10808389)
I’m a fan of OVH myself[1]. Heard good things about x4b.net as well.

[1] https://news.ycombinator.com/item?id=10807392

Re:Linode is suffering on-going DDoS attacks by otterley on Tuesday December 29, 2015 @02:24PM (#10808507)
If they're using the typical anti-DDoS mitigation equipment (usually Arbor) then they're almost certainly throwing out good traffic along with bad. AWS's approach of having an ingress pipe large enough to absorb DDoS traffic rates and allowing only valid HTTP requests to pass through the ELB is a superior solution, IMO.
Re:Linode is suffering on-going DDoS attacks by deno on Tuesday December 29, 2015 @02:43PM (#10808610)
Well, yeah, infinite resources are always the best DDoS solution ;)

Valid requests ≠ clean traffic. That just moves the attack couple of layers up.

Re:Linode is suffering on-going DDoS attacks by scurvy on Tuesday December 29, 2015 @12:05PM (#10807600)
After thinking about it more, I now realize that there is one way that AWS/GCE can lessen effects of a DDoS. Given that a lot of attack machines are hosted inside of AWS/GCE, they can handle this with lots of internal network bandwidth. They see less north/south (transit) attack bandwidth than a regular hoster would.

In short, if a large amount of the attack traffic originates inside of AWS/GCE, it's better to be off in AWS/GCE.

Between the two, neither have any kind of automated tooling to detect and shutdown rogue attack instances (AFAICT). They still rely on third parties to tell them "Hey, you're sending me 300 gbps of DoS traffic."

Re:Linode is suffering on-going DDoS attacks by eeeeeeeeeeeee on Friday January 01, 2016 @09:56PM (#10824736)
Sure, but tell me what your bill is at Amazon when you get hit by a massive attack. Amazon is incredibly resilient against attacks due to their size, but it comes at a price too.
Re:Linode is suffering on-going DDoS attacks by frik on Tuesday December 29, 2015 @12:29PM (#10807769)
you mean discussions like this (2 days ago), first comment sub-thread: https://news.ycombinator.com/item?id=10794951
Re:Linode is suffering on-going DDoS attacks by thrownaway2424 on Tuesday December 29, 2015 @12:38PM (#10807847)
Exactamente.
Re:Linode is suffering on-going DDoS attacks by Diederich on Tuesday December 29, 2015 @11:13AM (#10807324)
That is true.

However, how many of the people impacted by the current DDoS against Linode are only affected BECAUSE they are using Linode?

Re:Linode is suffering on-going DDoS attacks by busterarm on Tuesday December 29, 2015 @11:24AM (#10807399)
Guilty.

This has caused all kinds of pain for us this weekend. We use WPEngine to host some sites, who in turn host everything on Linode.

Honestly WPEngine has some real nerve charging people big bucks for a failover plan that apparently doesn't exist. This is just another of a half-dozen or so Linode failures that took us and loads of other of their customers down completely. We're lucky that we planned for this ahead of time, but we weren't 100% ready to go live on a competing service either. A lot of folks are working on their vacations right now.

I think the two questions I'll be asking every host now and into the future are:

1) Do you host your services on Linode. 2) If you do, do you failover to another provider?

A yes to the first question and a no to the second is a non-starter in my experience.

Re:Linode is suffering on-going DDoS attacks by davidvwp on Wednesday December 30, 2015 @10:56AM (#10812522)
Hey Busterarm,

David here from WPE. Are you using our HA solution (Geographic redundancy)? Did that fail? If so, did you open a ticket with support & inquire about an SLA credit?

Keep in mind we have many levels of redundancy with all of our plans, but not every plan includes Geo redundancy. Very few sites anywhere truly use hot/hot geo redundancy because of the complexity of database syncing and the expense of duplicating server environments in different data centers.

In a DDoS attack (which can happen at any data center, backbone provider, etc.) or any other data center wide outage, the only work around is Geo redundancy. Many hosts have different infrastructure providers and offsite redundancy (e.g. all of our customers enjoy offsite backups), which can allow you to recover your site even if the data center burns down, but don't necessarily provide a true hot/hot level of redundancy.

A true hot/hot configuration requires hardware in multiple locations, live database syncing, and geo load balancing. While some of our customers do purchase geo redundancy, it isn't a default part of every account.

Again, if you're an HA customer and that failed, please open a ticket. If you have questions on Geo redundancy, you can ask about it in that ticket as well. If you do move hosts, remember any host can be the victim of a DDoS attack and unless you have true hot/hot or hot/warm geo redundancy with your account, you could still be susceptible to data center wide outages like large-scale DDoS attacks.

-D

Re:Linode is suffering on-going DDoS attacks by workitout on Tuesday December 29, 2015 @07:18PM (#10809986)
And yet after years on Linode they are still the best out of many I've tried the past 2 decades, weird how our experiences vary isn't it?
Re:Linode is suffering on-going DDoS attacks by busterarm on Tuesday December 29, 2015 @08:47PM (#10810265)
The entire market is quite bad across the board.

I guess what I should be really saying is that if you're hosting your platform on these services and offer some kind of redundancy, make that redundancy through another host.