News for hackers. Stuff that matters.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by spindritf on Thursday February 13, 2014 @08:28AM (#7231192)
Somewhat ironically, the large French hosting provider OVH was one of the largest sources of our attack and also a victim of a large scale NTP amplification attack around the same time.

And their own semi-official ntp server supports monlist with a hefty response

    $ ntpdc -c monlist ntp0.ovh.net

    remote address          port local address      count m ver rstr avgint  lstint
    ===============================================================================
    10.x.x.246             123 213.251.128.249      515 3 2      0     12       0
    10.x.x.248             123 213.251.128.249      396 3 2      0      7       0
    10.x.x.245             123 213.251.128.249      104 3 2      0     10       0
    212-x-x-101.rev.pon   123 213.251.128.249   178326 3 4      0      0       0
    sw.178.x.x.248-n5.f   123 213.251.128.249       12 3 2      0     12       0
    proxy.ovh.net          46863 213.251.128.249   252113 3 3      0      0       0
    v1.ovh.net             50733 213.251.128.249     2443 3 3      0      0       0
    a2.ovh.net             44965 213.251.128.249  3394192 3 3      0      0       0
    cross.rfid.ovh.net     33352 213.251.128.249    11823 3 3      0      0       0
    10.x.x.176             123 213.251.128.249     1865 3 2      0      4       0
    gw6.ovh.net              123 213.251.128.249     1476 3 4      0      3       0
    sw.5.x.x.248-n5.fr   123 213.251.128.249      361 3 2      0      2       0
    b6.ovh.net             40862 213.251.128.249     1095 3 3      0      4       0
    10.x.x.245             123 213.251.128.249      164 3 2      0      6       0
    10.x.x.211            123 213.251.128.249   314567 3 2      0      4       0
    .
    .
    .
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by wjoe on Thursday February 13, 2014 @08:45AM (#7231280)
I have a server hosted with OVH, they actually sent me a message a week or so ago advising me my server running a vulnerable version of NTP so that I could update it. I think they were even going to update it for me, but I went ahead and updated it myself anyway.

This was at least a week before the news of the big DDoS attack this week, so I'm surprised their own servers still had the vulnerable config/versions.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by vidarh on Thursday February 13, 2014 @10:46AM (#7231922)
I have a server with OVH, but frankly I'm considering moving elsewhere after we've now been repeatedly hit by DOS from servers at OVH. It's fairly low grade, primitive SYN-flood attack that we easily knock back within minutes each time the attacker moves elsewhere (clearly he does not have access to many server resources, or he might have actually managed to muster enough simultaneous resources to do some damage; he's right this minutes wasting resources getting a SYN-flood from some no-name Russian hosting provider dropped by our firewall at a low enough rate that I can keep an eye on it live with tcpdump).

But while our colo provider was extremely responsive and started calling OVH and the other providers right away, and I also emailed evidence to OVH repeatedly, we were met with total silence. The other providers used reacted quickly. OVH let the servers continue to hammer us for days.

I'm seriously considering just dropping all their net blocks in our firewalls. We have next to no legitimate traffic originating there anyway.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by thaumaturgy on Thursday February 13, 2014 @02:17PM (#7233540)
FWIW large portions of ovh.com have been blackholed on my web server for persistent Wordpress comment spam to hosted sites.

Unfortunately, I can't blackhole their traffic on mail services, because IIRC some open source mailing lists use them.

OVH is not my favorite network.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by dfc on Thursday February 13, 2014 @10:59AM (#7232013)
They either turned it off or you are on the right side of a firewall ruleset.

PS: Why did you "X.X" out IPs from a RFC1918 address space?

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by vidarh on Thursday February 13, 2014 @11:05AM (#7232050)
Well, we're definitively not on s net block that OVH ought to give access to monlist:

ntpdc -c monlist ntp0.ovh.net | wc -l 602

Yikes.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by jebblue on Thursday February 13, 2014 @06:28PM (#7235476)
Snort has detected the following against my server lately:

  EXPLOIT ntpdx overflow attempt
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by bhauer on Thursday February 13, 2014 @10:12AM (#7231738)
Great write-up and very helpful for those of us who, despite doing so for years, remain amateurs at running our own servers. I am among those who think the Internet would be better as a whole if more people did in fact run servers—server software would gradually become easier for us amateurs to install and run without leaving it in a state that is open to nefarious exploits. But for the time being, I appreciate it when experts take the time to explain simple counter-measures as you have done. Thank you!

As far as I am aware, I am not responsible for any Internet-facing NTP servers (I certainly never set one up willingly), but it's good to have this in the back of my mind now in the off-chance that I ever do set one up.

I did have one of my Windows machines used for DNS amplification. I wrote about the incident [1] at my blog because I had been a bit surprised that it was not sufficient to simply disable recursion. That much had seemed like common sense, and I thought I had been so clever and thorough in turning it off. But later I found attackers were leveraging my server's willingness to provide a list of root DNS servers in response, even with recursion disabled. I ended up deleting the list of root servers and the problem went away. (Though, to be clear, I never ran the incident by any DNS experts, so I may have misdiagnosed the whole thing.)

I don't know what else I don't know about amplification attacks, so reports such as yours are helpful for people like myself who find it fun to run our own servers, but don't consider it an area of expertise.

[1] http://tiamat.tsotech.com/dns-amplification

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by asdf1011 on Friday February 14, 2014 @01:07AM (#7237203)
We have some racks with public facing ILOM interfaces which sit outside the firewall, which turns out have ntpd running. We only noticed when our international bandwidth crawled to a halt due to them being used in an NTP attack.

It's a hassle, as they're old machines and out of support contract (so we can't upgrade the firmware), and so far as I can tell there's no way to turn off public access to ntpd over the admin interfaces. We're stuck with having to go to the hosting company and change the cabling to route them through the firewall.

Just because you didn't set up ntpd doesn't mean you don't have it running (somewhere).

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by rhizome on Thursday February 13, 2014 @02:04PM (#7233431)
Not knowing about this particular vulnerability doesn't make one a pejorative amateur.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by bhauer on Thursday February 13, 2014 @03:53PM (#7234247)
I am part of the group I was referring to as amateurs, and I was not using the word in a pejorative manner. I mean it literally. We, the amateurs I am speaking of, are not experts at server administration and do it either out of necessity, for fun, or out of principal. For me, it's a bit of fun and principle—I feel it's a good thing for me to more or less know how to manage a server even as I acknowledge that experts know much more about it than I.

I would like to see servers demystified in general, and that's why I applaud articles such as this one that make the necessary counter-measures/workarounds plain and clear for those like myself.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by rhizome on Thursday February 13, 2014 @06:49PM (#7235605)
That's all well and good, and I wish you luck in your ambitions, but you're still making a category error.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by bhauer on Thursday February 13, 2014 @08:42PM (#7236282)
I'm reaching to understand the point you're making.

I assume you mean that you interpret what I've said as implying anyone who does not know about NTP amplification attacks is an amateur server administrator. I do not mean to imply that. I simply mean that for people like myself, who are in fact amateur server administrators, this sort of down-to-earth style of article (here is what the problem is; here is how to deal with it) is very welcome.

Although it's only academic at this point, I am curious why you think I used the word "amateur" pejoratively and what category you believe I am describing in error. I ask simply because I'd like to avoid making similar errors in the future.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by rhizome on Thursday February 13, 2014 @09:49PM (#7236549)
Because you more-than-implied that nobody who got hit with it is a professional.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by aidos on Thursday February 13, 2014 @09:10AM (#7231423)
As far as I can tell these attacks always rely on amplification using IP Spoofing. I take it there's no way of mitigating that in a lower layer without adding some leaky abstraction or general overhead to the network? So, for example, (speaking as someone who knows nothing about these things) you could add some sort of handshake along the lines of:

    ntp server sees request from 1.1.1.1 (spoofed by attacker)
    ntp server goes to 1.1.1.1 to check that they really sent the request (sort of ack type thing)
    1.1.1.1 comes back to say that it's an uninitiated request
    ntp server discards similar future requests for some time
Obviously that would require more toing and froing, along with more white / black list tracking etc. Then again, can't all machines have sensible defaults in their firewalls to stop them from participating in such attacks?

Is this not an issue for TCP?

EDIT: I'm assuming it's because UDP doesn't do any checking / acknowledge stuff by default?

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by devicenull on Thursday February 13, 2014 @09:13AM (#7231446)
Yes. And that's exactly what's been implemented in modern versions of ntpd (> 4.2.7p26, 2010/04/24... yes, 2010).

The problem is: 1) No one has upgraded NTPD (and often can't, for embedded devices like IPMI controllers) 2) This can be fixed by basic configuration in older NTPD versions, but up until recently many linux distributions were shipping vulnerable configs.

This particular command (monlist) is a management query, it's in no way related to serving up accurate time.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by michaelmior on Thursday February 13, 2014 @10:04AM (#7231696)
As far as I can tell, the default Ubuntu ntpd config still allows monlist. (That is, /etc/ntp.conf in saucy doesn't have disable monitor.)
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by makomk on Thursday February 13, 2014 @11:44AM (#7232321)
The Debian default config has noquery set for everywhere but localhost which is I believe safer (disables all ntpdc queries not just the one used here). Ubuntu's likely the same.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by michaelmior on Thursday February 13, 2014 @12:05PM (#7232483)
The default Ubuntu config has the following

  restrict -4 default kod notrap nomodify nopeer noquery                          
  restrict -6 default kod notrap nomodify nopeer noquery
Edit: Note that this also exists in Ubuntu 12.04, so the latest LTS is fine as well.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by michaelmior on Thursday February 13, 2014 @11:49AM (#7232360)
Yes, noquery is enabled (at least in saucy). It isn't however in precise, which is the current LTS. I'm unclear of the difference between noquery and disable monitor.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by devicenull on Thursday February 13, 2014 @11:19AM (#7232150)
disable monitor is not really the best way to fix this. You're better off adding 'noquery' to the end of the 'restrict default' lines.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by michaelmior on Thursday February 13, 2014 @11:49AM (#7232364)
Thanks for clarifying. I'm not familiar with ntpd configuration as it's something I usually just install and forget about.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by dfc on Thursday February 13, 2014 @11:13AM (#7232108)
What are the default restrictions? Is noquery enabled?
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by the_mitsuhiko on Thursday February 13, 2014 @09:13AM (#7231447)
UDP is a message. It's the same as sending a letter with the sender's address on. You can lie about your own address, but if you want the other person to send you a reply it's better to write the correct information on.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by corresation on Thursday February 13, 2014 @09:15AM (#7231462)
TCP establishes a two way connection (SYN, SYN-ACK, ACK), so you can send the original SYN but the SYN-ACK will go to someone else and be discarded. UDP is fire and forget, in contrast.

There is a proposal from 2000 that is mentioned in the article (http://tools.ietf.org/html/rfc2827) that recommends that source networks filter out originating traffic that isn't legitimate. It is being implemented slowly.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by yread on Thursday February 13, 2014 @07:16AM (#7230883)
Details on the SNMP amplification

http://www.nothink.org/misc/snmp_reflected.php

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by voltagex_ on Thursday February 13, 2014 @09:00AM (#7231368)
Sigh. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=733940

FWIW, if you install the ntp package and do ntpdc -n -c monlist localhost you'll get a response but I haven't checked if it's configured by default to reject non-LAN requests.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by alex1 on Thursday February 13, 2014 @09:22AM (#7231493)
FWIW, here's what I got on an Ubuntu 12.04.3 server running on my LAN. It looks like we should be fine with the defaults on Ubuntu at least. (Obviously always a good idea to use ufw/iptables to block everything you don't need exposed so you don't have to worry about stuff like this).

Before installing ntp (from another host on my LAN):

  $ ntpdc -n -c monlist 192.168.1.50
  ntpdc: read: Connection refused
After installing ntp (from another host on my LAN):

  $ ntpdc -n -c monlist 192.168.1.50
  192.168.1.50: timed out, nothing received
  ***Request timed out
After installing ntp (from the server itself):

  $ ntpdc -n -c monlist localhost
  remote address          port local address      count m ver rstr avgint  lstint
  ===============================================================================
  91.189.94.4              123 192.168.1.50         1 4 4    1d0     54      54
  ...
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by voltagex_ on Thursday February 13, 2014 @02:59PM (#7233850)
Thanks, I should have checked this.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by devicenull on Thursday February 13, 2014 @09:15AM (#7231459)
Debian's something I checked when I first started seeing this, and their default config is not vulnerable.

Being able tod this via localhost is not a problem, it's when it's open to the internet.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by shawabawa3 on Thursday February 13, 2014 @09:12AM (#7231434)
Just tried it on my server. Works for localhost but times out remotely
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by powertower on Thursday February 13, 2014 @08:54AM (#7231323)
> it returns a list of up to the last 600 IP addresses that last accessed the NTP server

It just gives up that data to anyone that asks? Seems like a huge privacy issue.

Imaging Apache or Nginx giving up the last 600 IPs it served and maybe the URLs they went to.

edit: there is always the occasionally open Apache /server-status handler that leaks this type of data.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by Karunamon on Thursday February 13, 2014 @11:19AM (#7232147)
>It just gives up that data to anyone that asks? Seems like a huge privacy issue.

I could do a nmap on the public internet and probably get a similar amount of addresses. An IP is about as "private information" as a phone number nowadays (You know, those things that get sent out en masse in yellow and white books for public consumption with real-life names next to them).

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by buzzkills on Thursday February 13, 2014 @10:06AM (#7231703)
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by devicenull on Thursday February 13, 2014 @01:22PM (#7233055)
Huh? Those are completely different issues.

So you synced your time with a server. Why would anyone else care about that? Why does it matter if someone else knows you're syncing time?

This is a very different service then a web browser.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by NelsonMinar on Thursday February 13, 2014 @08:17PM (#7236143)
Some historical context on NTP monlist: it's an old debugging interface from the days of the Friendly Internet, when services were more open and people were much less worried about this kind of security. NTP daemons give up a whole lot of information if you ask them; see also the "peers" and "sysinfo" commands, for instance.

Back in 1999 I used these monitoring commands to spider the NTP network, surveying some 175,000 hosts from a desktop workstation. Lots of fun! This kind of survey is much harder to do now because so many systems are locked down. http://alumni.media.mit.edu/~nelson/research/ntp-survey99/

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by tinco on Thursday February 13, 2014 @08:19AM (#7231143)
This is offtopic, but this post was deleted some time ago, and now it's back, it was submitted by jgrahamc then also.

What happened to it? Did the algorithm snip it, but did jgrahamc undelete it somehow, or a mod? Just curious about the way those things work, not complaining.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by jgrahamc on Thursday February 13, 2014 @08:28AM (#7231189)
I deleted it and resubmitted it later.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by rahimnathwani on Thursday February 13, 2014 @07:15AM (#7230879)
Is the response to MONLIST also sent as UDP? If so, why does CloudFlare even accept those packets to IP addresses used for web hosting? Shouldn't all legitimate traffic be TCP on ports 80 and 443?
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by mnw21cam on Thursday February 13, 2014 @07:32AM (#7230937)
The packets have to actually reach you in order for you to filter them out. If you have a 300Gbps incoming pipe, and you're getting 300Gbps of attack traffic, then there isn't any space left in the pipe for your legitimate traffic. It doesn't matter that your router is throwing away the packets as soon as it receives them.

Also, web servers might want to consult NTP servers now and again.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by alex1 on Thursday February 13, 2014 @07:55AM (#7231019)
> Also, web servers might want to consult NTP servers now and again.

CloudFlare doesn't host web servers for their customers. They forward HTTP/HTTPS requests to origin servers outside of their network (or serve from their cache). I don't think the DDoS traffic actually hit any of their customers' origin servers (assuming origin server IPs are not known by the attackers). But yeah, it still means CloudFlare's incoming pipes being hit with 400Gbps of traffic before they're able to filter anything.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by omh on Thursday February 13, 2014 @07:30AM (#7230934)
By the time they get the chance to reject the packet it's too late. At that point it's already on their network and it's consumed their bandwidth.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by andyjohnson0 on Thursday February 13, 2014 @09:31AM (#7231543)
UDP allows source and destination ports to be specified separately [1], so the attacker could just spoof the source port by setting it to 80 or 443. Unless the NTP server was specifically configured to not reply to well-known port numbers [2], this would result in the reply going to the spoofed port.

[1] http://en.wikipedia.org/wiki/User_Datagram_Protocol#Packet_s...

[2] Not sure if this is even possible

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by noselasd on Thursday February 13, 2014 @03:48PM (#7234210)
Sure. That's what they do.

The problem is you have UDP packets on port 123 coming from all over the world hammering at your door. They're politely getting dropped by firewalls, but they consume all the bandwidth in to the edge of your network, so the legitimate traffic cant get through.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by poobrains on Thursday February 13, 2014 @08:17AM (#7231125)
How do you test a server for this attack? I want to make sure my servers don't participate.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by biot on Thursday February 13, 2014 @08:58AM (#7231354)
The fine article states:

  You can check whether there are open NTP servers that
  support the MONLIST command running on your network by
  visiting the Open NTP Project[0]. Even if you don't think
  you're running an NTP server, you should check your
  network because you may be running one inadvertently.
[0] links to http://openntpproject.org/
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by e12e on Thursday February 13, 2014 @10:50AM (#7231939)
As I happen to have openntpd installed on a box I attempted to test this from (in Debian that package conflicts with ntp -- which includes the ntpdc client) -- I also found this:

https://github.com/sensepost/ntp_monlist

It at least correctly identifies ntp0.ovh.net as responding -- and seems to match up with what openntpproject.org thinks...

[edit: apparently this (partly) also illustrates why more people should heed the advice to "run only what you need, listen only where you must" -- or in other words, make sure that:

    netstat -lnutp # listening, numerical, udp, tcp, program
gives essentially no output, at the very least not a lot of 0.0.0.0:x (listening on all interfaces). I'm always a little sad when people don't check that, and just throw up some complicated iptables-rules -- before checking if they're actually running some daemons that should be removed, or pointed at less public interfaces.]
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by m-arnold on Thursday February 13, 2014 @08:28AM (#7231196)
You can use ntp client:

ntpdc -c monlist 1.2.3.4

For more info see my blog post (it is related to VMware ESXi but instructions are useful for any ntpd): http://ar0.me/blog/en/posts/2014/01/howto-prevent-malicious-...

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by devicenull on Thursday February 13, 2014 @09:16AM (#7231466)
Don't forget to test your IPMI controller as well, if you have that exposed to the internet!
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by duskwuff on Thursday February 13, 2014 @11:38AM (#7232286)
Those really shouldn't be exposed to the Internet, full stop.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by jlgaddis on Thursday February 13, 2014 @11:06AM (#7232057)

  $ nmap -sU -pU:123 -Pn -n --script=ntp-monlist <target>
Note that that only checks if the target responds to the monlist command.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by jadc on Thursday February 13, 2014 @08:28AM (#7231190)
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by unethical_ban on Thursday February 13, 2014 @03:39PM (#7234140)
What about a mesh network? How would they implement BCP38?

If a bunch of computers are hooked together in a mesh relaying traffic in all directions, how can one do ingress filtering?

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by mturmon on Thursday February 13, 2014 @11:32AM (#7232237)
OT, but: Tasteful choice of album cover art for the pic at the top.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by junto on Thursday February 13, 2014 @09:47AM (#7231617)
I'm not much of a network guy, but is it possible for Cloudflare to just redirect that DDOS traffic back to the NTP server that sent it?

This would have two benefits. Firstly the owner of the insecure NTP server is going to get a nasty message to fix their damn server, and secondly, the insecure NTP server gets taken out of the attack and becomes useless to the attacker.

As a visual reference, it would be a bit like in Star Wars when Mace Windu fights Palpatine on Coruscant [1]: http://www.youtube.com/watch?v=Pk4AiCnMqpg#t=2m35s

Eventually these server provider who have left their servers wide open will get the message when there NTP servers no longer respond?

1. http://starwars.wikia.com/wiki/Showdown_on_Coruscant

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by jgrahamc on Thursday February 13, 2014 @09:51AM (#7231635)
No, we are not going to do that.

Two reasons: it's a bad idea and it wouldn't help very much. Each individual NTP server is only generating a modest amount of traffic and such a response might well go unnoticed by the NTP server. Also, it would mean we'd have to generate 400Gbps back to the NTP servers creating an enormous amount of traffic.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by MichaelGG on Thursday February 13, 2014 @02:06PM (#7233453)
As someone who unwittingly participated in such attacks, that's a terrible idea. I had DNS/NTP servers setup on a helper box and didn't lock it down and forgot about it. It has has multiple 1Gbps uplinks and was sending a couple hundred Mbps of traffic out. If someone started sending a couple hundred Mbps of traffic back, first, I wouldn't notice until I got a bill later on. Second, if for some reason I happened to look at that machine and saw a huge amount of incoming traffic from a single source, I'd probably just ask for it to be null-routed. I might incidentally notice that I was sending out a lot of packets, maybe.

To actually get the benefits you're suggesting, CloudFlare would have to actually DDoS (spoof data from multiple IPs to the point of overwhelming the target) every participating NTP server. So on a 100Gbps attack, they'd need to send out, I dunno, 1Tbps of spoofed traffic? Not probably a wise idea.

The correct way is to just contact the network abuse team. That way I get an email telling me to fix my server. And it's taken care of as quickly as possible.

And the real correct way is for all ISPs to implement source filtering, but last I checked, that was laughably far from being implemented. Lots of ISPs would even take sources of private IPs and merrily forward them on.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by Florin_Andrei on Thursday February 13, 2014 @11:35AM (#7232258)
For tons of historical precedents explaining why this is a bad idea and never ends well, look up the concept of vendetta.

http://en.wikipedia.org/wiki/Feud

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by petermonsson on Thursday February 13, 2014 @10:27AM (#7231809)
This is probably a stupid question, why can't tier 1 providers (whom I suppose there are relatively few of and who I would expect to incorporate best practices) just decide to kill any NTP monlist UDP that ever crosses any of their NPUs?

Why would that not solve a large part of the problem?

Thank you in advance.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by jlgaddis on Thursday February 13, 2014 @11:04AM (#7232047)
I don't work for a Tier 1 ISP but I do work for an ISP.

As a customer, I don't want my ISP screwing with my traffic. As a provider, I don't want any customers complaining because we screw with their traffic.

To block monlist and only monlist queries, we'd have to be looking into the layer 7 payload of IP traffic. I'd rather not do that.

The brute-force method would be to block traffic to/from 123/UDP but that's gonna mess up a lot of stuff (including my own).

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by vidarh on Thursday February 13, 2014 @11:03AM (#7232039)
Cost. At typical backbone speeds there are problems enough dealing with basic routing at sufficient throughput already. "Nobody" at that level wants to also have to pattern match packets.

Cisco, Juniper etc. who manufacture high end routers would certainly love it.

A better alternative is to stop or limit source ip spoofing, because you can filter it on the interfaces connecting smaller providers and customers rather than the most resource-constrained routes to other backbone providers. And that's slowly happening (I'm saying, while looking at tcpdump output from a SYN attack that might very well use spoofed IPs). It's simpler because you "only" need a single lookup against a few bytes per packet per interface instead of potentially having a long list of patterns to check against the whole packet.

> and who I would expect to incorporate best practices

Don't bet on it. They will when it makes a big difference to them. But for many of these types of attack you'll see purely reactive reactions because it's often far cheaper (for them) vs. the costs of routing hardware etc. that can do enough processing per packet to be viable.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by kamkazemoose on Thursday February 13, 2014 @10:39AM (#7231866)
I'm not sure at what layer it would be visible that the packets are ntp monlists? The basic answer is probably that the providers just aren't filtering at that level, like if you need layer7 visibility, they don't want to do dpi on any packet that crosses their boundary. Even if you don't need L7 visibility, it would still require another element of processing and filtering that they might not be interested in doing since it will almost always result in some extra cost to run.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by unethical_ban on Thursday February 13, 2014 @03:38PM (#7234125)
Many people have said it and I'll emphasize: To the extent possible, ISPs should not be doing layer 7/deep packet inspection. They carry traffic. They shouldn't filter. Not only is it unethical, but it's impractical for providers.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by noselasd on Thursday February 13, 2014 @03:51PM (#7234228)
Stepping in and helping mitigate DDOS attacks such as this, by e.g. having the Tier-1s dropping traffic destined to the victim at their edges might be ok.

However, Tier 1 providers should not in any way police the internet.

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by erichocean on Thursday February 13, 2014 @10:33AM (#7231834)
That seems like a great idea, although it could become a slippery slope.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by peterwwillis on Thursday February 13, 2014 @04:27PM (#7234504)
So again we find the problem is protocols that are designed for convenience and not security. Sure, network providers could filter out bogus routes, but that's a band-aid more than a fix; the protocol is still broken from a security perspective. Nobody would stand for using rsh with host-based authentication in today's age, but for other protocols it's fine? And public services for the internet at large are great, until they become tools for the public to abuse other people. These protocols either need to be fixed to prevent abuse, or switch to using tcp (which nobody wants - so fix the protocols!)
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by TA-bye on Thursday February 13, 2014 @12:02PM (#7232459)
All NTP seriousness aside, this new "record-breaking" DDoS attack was only possible because CloudFlare -- after the Spamhaus attack -- upgraded and expanded their network endpoints all over the world. When the next attack hits and they have again upgraded their connections with 100Gb/s combined, they'll be able to say that there was again a new record, this time it was 500Gb/s.
Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by JulianMorrison on Thursday February 13, 2014 @11:36AM (#7232268)
I wonder, is QUIC vulnerable?

UDP? [✓]

Amplification? [✓]

Spoofable? [?]

Re:Technical Details Behind a 400Gbps NTP Amplification DDoS... by oasisbob on Thursday February 13, 2014 @01:20PM (#7233036)
QUIC datagrams should be as spoofable as anything else using UDP.

The _QUIC Crypto_ design doc contains a section that covers spoofing [1], and seems to push responsibility for DDoS mitigation to the server implementation:

"[...] servers may decide to relax source address restrictions dynamically. One can imagine a server that tracks the number of requests coming from different IP addresses and only demands source-address tokens when the count of “unrequited” connections exceeds a limit globally, or for a certain IP range. This may well be effective but it’s unclear whether this is globally stable. If a large number of QUIC servers implemented this strategy then a substantial mirror DDoS attack may be split across them such that the attack threshold wasn’t reached by any one server."

[1] https://docs.google.com/document/d/1g5nIXAIkN_Y-7XJW5K45IblH...