News for hackers. Stuff that matters.
Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @11:43AM (#5613529)
Disclaimer: VideoLAN president and lead VLC developer here.

The attack started on our new mirroring system (powered by mirrorbrain) 2.5 days ago, during the night (after 2am, so we were sleeping).

We were woken up (OP and I) in the morning by many mirrors complaining of high bandwidth use. The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.

Afterwards, North America got up, and things got worse. We had up to 1660 req/s, so around 292 Gbps...

This is very weird for a DDoS, to be honest.

Our front machine that splits the down mirrors was taking most of the load, and we were able to find the patterns to drop the botnet connexions, in order to not kill too much our mirrors. I won't discuss too much of the patterns, as you can imagine, but as usual, I'll be happy to discuss it IRL or by mail.

Tweaking the front server was also important to reduce the number of open connexions, to not kill our server.

2.5 days after, the attack is still going on, with an average of 500req/s.

The video was done using logstalgia, using scripts of OP, on my machine (<troll>he was running eclipse, he couldn't do both at the same time :)</troll>).

Re:This is what a DDoS attack looks like by derefr on Friday April 26, 2013 @06:25PM (#5615862)
Is there a reason VLC doesn't offer a Bittorrent download option, by the way? As far as I know VLC's audience is pretty technologically-inclined, so they would understand what a magnet link is, and you'd be able to just switch off the archive-download option for a while in cases like this. (As a bonus, if you embed a magnet link in your page, you're implicitly embedding a SHA-verification of the resulting torrent file.)
Re:This is what a DDoS attack looks like by subsystem on Friday April 26, 2013 @12:04PM (#5613641)
"This is very weird for a DDoS, to be honest."

Maybe it was aimed at bandwidth costs?

Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @12:11PM (#5613668)
Maybe. Seriously, no idea.
Re:This is what a DDoS attack looks like by drivebyacct2 on Saturday April 27, 2013 @04:11AM (#5617157)
That's precisely what I don't get in this case, especially. The source is out there, there are other people mirroring vlc. So if you're trying to censor it, you're going to fail pretty hard. In the absence of that, what can it be?

Of course! It must be the mplayer folks!!

Re:This is what a DDoS attack looks like by kin3tic on Saturday April 27, 2013 @04:15AM (#5617159)
Or in this case, who you'd literally least suspect: wmplayer. All hail wmv!
Re:This is what a DDoS attack looks like by vabmit on Friday April 26, 2013 @12:40PM (#5613846)
I don't know what request throttling solutions are available for nginx. But, for apache there are modules and scripts/code that can rate limit traffic and requests from specific IPs. Depending on the size of the botnet attacking, rate limiting requests by IP or network may have been a way to mitigate the DDoS effects. Again, depending on the specifics, you should be able to use iptables (if you're running Linux) to limit requests per source IP or IP range. The type of actions you can take with iptables are really quite robust.

I've had to deploy request rate limiting solutions in the past because of well meaning federated search mash-ups that included content from servers that I run.

Filtering by useragent seems to be a temporary solution. I think that's especially true since you've disclosed publicly that's what you're doing.

Re:This is what a DDoS attack looks like by daemon13 on Friday April 26, 2013 @03:58PM (#5615083)
With nginx you can easily rate-limit connections, requests and downloads (which you can throttle also). Can also do this by GeoIP - country, etc.

No need for extra steps - this module shall be compiled and part of nginx-full package from nginx official PPA, if OP is using Ubuntu.

Re:This is what a DDoS attack looks like by rb12345 on Friday April 26, 2013 @03:26PM (#5614918)
I had problems with the Windows updater apparently downloading version 2.0.6 and then not installing it. I ended up downloading and installing it manually instead. Could that be related somehow?
Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @07:41PM (#5616131)
VLC Windows updater?
Re:This is what a DDoS attack looks like by rb12345 on Saturday April 27, 2013 @07:19PM (#5619421)
Yes, the VLC Windows updater, which runs when VLC starts.
Re:This is what a DDoS attack looks like by jbk on Sunday April 28, 2013 @05:56AM (#5620773)
It has a very specific user agent, and does not call with this pattern.
Re:This is what a DDoS attack looks like by merlincorey on Friday April 26, 2013 @10:48PM (#5616620)
Why do you return 403 (according to the article) instead of 444? 444 will drop and close the connection, so I am guessing it has something to do with keep alives vs connection overhead?
Re:This is what a DDoS attack looks like by huhtenberg on Friday April 26, 2013 @01:30PM (#5614190)
Who toes have you stepped on recently, I wonder?

Also, pattern-wise - most of the traffic in the video seems to be hitting a single .asc file.

Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @01:50PM (#5614330)
No idea, seriously. VLC scammers, mainly.

The .asc is a logstalgia visualisation bug, because in fact, it merges the few .exe.asc requests with the massive .exe requests

Re:This is what a DDoS attack looks like by newman314 on Friday April 26, 2013 @01:21PM (#5614121)
What did you tweak in nginx?
Re:This is what a DDoS attack looks like by daemon13 on Friday April 26, 2013 @03:59PM (#5615088)
Is this originating from specific country or all over?
Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @07:42PM (#5616133)
All over.
Re:This is what a DDoS attack looks like by to3m on Friday April 26, 2013 @01:08PM (#5614026)
"Disclaimer"?
Re:This is what a DDoS attack looks like by Ideka on Friday April 26, 2013 @03:23PM (#5614899)
Looks more like a "claimer" to me.
Re:This is what a DDoS attack looks like by mrtbld on Friday April 26, 2013 @04:49PM (#5615384)
He wanted to write "disclosure" I guess. (He's not a native English speaker.)
Re:This is what a DDoS attack looks like by eterps on Friday April 26, 2013 @10:55AM (#5613234)
Re:This is what a DDoS attack looks like by scoot on Saturday April 27, 2013 @08:54AM (#5617633)
If you like logstalgia, you'll love gource[1] which visualises source control change history. It's also available as a brew recipe on the Mac for easy-peasy install.

[1] https://code.google.com/p/gource/

Re:This is what a DDoS attack looks like by rozap on Friday April 26, 2013 @12:44PM (#5613869)
This has pretty limited use, but holy shit is it awesome. Thanks!
Re:This is what a DDoS attack looks like by onemorepassword on Friday April 26, 2013 @03:21PM (#5614894)
Limited use?

As someone who over the past decade regularly had to sell higher management on investing in infrastructure, I can tell you this will be extremely useful.

Seriously, even if it's just a random visualization of some normal peek traffic, it will go down easier than dozens of actual arguments, slideshows with bullet points or fat reports nobody reads. Show the video, do a bit of handwaving and there's your budget...

For any techie trying to convince their boss they need more servers, this is a very powerful tool. Use with caution.

Re:This is what a DDoS attack looks like by rozap on Friday April 26, 2013 @03:34PM (#5614955)
So I installed the project and ran it with my log files and I'd like to retract my previous claim of it not being useful. You're right, I can definitely see the use here, it puts stuff into perpective quite well.

It's easy to write off shiny visualizations as silly (which is what I did above) but they definitely offer a perpective that you can't get any other way.

Re:This is what a DDoS attack looks like by BaconJuice on Friday April 26, 2013 @11:25AM (#5613412)
Thank you.
Re:This is what a DDoS attack looks like by 11031a on Friday April 26, 2013 @11:55AM (#5613594)
awesome find. thanks
Re:This is what a DDoS attack looks like by est on Friday April 26, 2013 @08:50PM (#5616339)
Damn that's impressive. Someone please port a realtime WebGL version!
Re:This is what a DDoS attack looks like by Kekeli on Tuesday April 30, 2013 @01:44PM (#5633743)
Love this
Re:This is what a DDoS attack looks like by onemorepassword on Friday April 26, 2013 @03:13PM (#5614853)
All I could think of was "this is what a DDoS attack will look like in Hollywood products for the next decade".

Awesome visualization.

Re:This is what a DDoS attack looks like by GhotiFish on Saturday April 27, 2013 @12:18AM (#5616792)
I was just thinking, there weren't nearly enough laser sounds, polygons, or explosions.

It's not quiet there yet!

Re:This is what a DDoS attack looks like by dlhavema on Friday April 26, 2013 @06:51PM (#5615960)
oh this isn't fantastic enough for CSI or BONES to use... not enough 3D graphics... very cool visual none the less..
Re:This is what a DDoS attack looks like by x0054 on Friday April 26, 2013 @01:45PM (#5614296)
Perhaps it's worth it to code a quick and dirty solution using JavaScript encryption. On your download page setup a script that would receive a given encrypted string, decrypt it with a provided key, and the use it to prepend to the download link. On the server, symbolically link the file on demand and send it to only one user, ip limited. This way the attack, though still can be automated, would require some code rewrite from that attacker, which might be beyond his/her abilities. Also, if the encryption algorithm is CPU intensive, then it wold require several seconds of CPU time per request from the attacker.

To make the decryption CPU intensive you may simply use any encryption algorithm you like, many are available as JS libraries, but instead of giving the entire decrypt key, skip the last 2 digits, and let the end user brut force the last 2 digits in the client via JS. That way there is a computational cost to each attack request.

Just some ideas off the top of my head. Not sure at the moment how to implement the server side part at the moment, but I am guessing that their are server side rules that allow you to easily set per ip access restrictions to folders or fils.

PS: please excuse spelling, typing this on my iPhone.

Re:This is what a DDoS attack looks like by negativity on Friday April 26, 2013 @11:41AM (#5613515)
I'd be willing to go out on a limb and estimate that maybe some private interests in Hollywood, with certain four letter acronyms, despise open source media player projects like VLC, since they might represent a channel that can potentially enable bypasses that can circumvent precious, precious DRM.

The perception being: if you can see the source of a media player program, the encryption might be implicitly compromised. This is a silly idea though, because it neglects certain realities about the very nature of electronic encryption, and media consumption. Maybe having source code lowers the bar in some respects, but the reality is that determined people will simply bootleg media anyway, by other means.

Not an accusation though, just that my tinfoil hat is tingling. Who else might be so motivated to attack an awesome software project like VLC?

Re:This is what a DDoS attack looks like by omni on Friday April 26, 2013 @11:59AM (#5613618)
Cool / "good" projects get attacked all the time. I suspect this is because the attacker is guaranteed an audience on HN to marvel in the results of their work. I think this is probably simpler and more likely than the tinfoil hat theory.
Re:This is what a DDoS attack looks like by Tobu on Friday April 26, 2013 @07:44PM (#5616139)
Another option is some shady people bundling VLC with spyware, some of which have been successfully clamped down: http://blog.l0cal.com/2011/07/07/these-companies-that-mislea...

Or, less likely but more timely, the GPL→LGPL relicensing to get an AppStore port, or the consultation to get the French DRM police to play its role in enforcing a conflicting law about interoperability.

Re:This is what a DDoS attack looks like by pavel_lishin on Friday April 26, 2013 @12:40PM (#5613848)
> Who else might be so motivated to attack an awesome software project like VLC?

Anyone who wants to show off a proof-of-concept attack to potential customers, without stepping on any huge toes. (e.g., it's relatively safe to DDoS VLC, not so much a U.S. government agency or a huge multi-national company.)

Re:This is what a DDoS attack looks like by snowwrestler on Friday April 26, 2013 @10:40PM (#5616588)
Why DDOS Reddit? Why DDOS Wikipedia? Why DDOS the New York Public Library website??

Within the last year or two I've given up trying to find reasons for DDOS attacks. I think of them now like Internet traffic weather. They just happen at random times and you better be prepared.

Re:This is what a DDoS attack looks like by AUmrysh on Friday April 26, 2013 @01:06PM (#5614005)
I could see it being some sort of message as CISPA was stalled in the Senate yesterday. More likely, however, is that we'll never know why.
Re:This is what a DDoS attack looks like by GigabyteCoin on Friday April 26, 2013 @01:24PM (#5614135)
I would suspect DDoS attacks to be akin to terrorist attacks in that somebody typically claims responsibility for them.
Re:This is what a DDoS attack looks like by snowwrestler on Friday April 26, 2013 @10:42PM (#5616597)
DDOS attacks are way more common than terrorist attacks, and there's a rarely a clue why they happen.
Re:This is what a DDoS attack looks like by Thaxll on Friday April 26, 2013 @11:36AM (#5613478)
What kind of person would DDoS a video player website?
Re:This is what a DDoS attack looks like by buro9 on Friday April 26, 2013 @11:41AM (#5613516)
Maybe Videolan isn't the target and the DDoS is against someone else in the form of using Videolan's bandwidth against someone else?

That's my general assumption of any attack in which the response to a request is in the many MB range and the request is in the bytes range.

That someone with limited bandwidth and many connections is attempting to acquire a large amount of bandwidth to attack someone else.

Re:This is what a DDoS attack looks like by JoshTriplett on Friday April 26, 2013 @12:07PM (#5613657)
Good thought, but source spoofing for an amplification attack wouldn't work here; from https://news.ycombinator.com/item?id=5613529 :

> The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.

Source spoofing would not allow downloading the whole file; the spoofed source address would get the first response packet and send an RST ("stop, no connection associated with this packet, go away") long before that point.

Re:This is what a DDoS attack looks like by skorgu on Friday April 26, 2013 @12:02PM (#5613633)
HTTP is over tcp though so it should be Hard to convince vlc's servers to reply to a third party. Seems like a silly ddos actually but what do I know.
Re:This is what a DDoS attack looks like by stackcollision on Friday April 26, 2013 @11:38AM (#5613500)
The same kind of person who would steal books from a library, or talk in a movie theater.
Re:This is what a DDoS attack looks like by aidenn0 on Friday April 26, 2013 @12:46PM (#5613885)
It it possible this is an accidental DDoS? VLC is popular to bundle with things, and all it would take is the code that checks for a new version and automatically downloads to have a bug that it always thinks there's a new version...
Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @01:57PM (#5614377)
This is impossible, seriously. I cannot share the details, but the HTTP headers shows clearly that this is not that case.

And more than 10 millions downloads per day?

Re:This is what a DDoS attack looks like by mortenlarsen on Saturday April 27, 2013 @12:21AM (#5616798)
Ubuntu downloads from it's own mirrors (hundreds of them), with very few exceptions for proprietary stuff that can't be distributed like Adobe Flash and ttf-mscorefonts-installer (not default). Also it would not download a .exe to install VLC.
Re:This is what a DDoS attack looks like by dave809 on Friday April 26, 2013 @01:30PM (#5614189)
Ubuntu 13.04 came out yesterday, the people upgrading might be reinstalling all there programs
Re:This is what a DDoS attack looks like by jbk on Friday April 26, 2013 @01:57PM (#5614382)
Ubuntu does not download a .exe...
Re:This is what a DDoS attack looks like by asperous on Friday April 26, 2013 @01:51PM (#5614334)
*their

I highly doubt 500 different people are upgrading ubuntu every second, but a bug in the software is possible I suppose.

Re:This is what a DDoS attack looks like by pan69 on Friday April 26, 2013 @04:33PM (#5615287)
And the bug just went away?
Re:This is what a DDoS attack looks like by clone1018 on Friday April 26, 2013 @10:59AM (#5613268)
I actually have Logstalgia running with my primary server for Minotar, and at 4,000 requests per second this is normally what it looks like. Awesome program!
Re:This is what a DDoS attack looks like by xfs on Friday April 26, 2013 @11:24AM (#5613399)
How does DDoS mitigation look like? Do they use realtime dashboard with similar visualization to cut off hotspots?
Re:This is what a DDoS attack looks like by WestCoastJustin on Friday April 26, 2013 @11:31AM (#5613451)
In my experience it is fairly manual. Here it is at a very high level. First you want to determine if this is really a DDoS or legitimate traffic.

You might be notified via downtime, alerts of load, in this instance, I suspect; download graphs, log analytic (if using a cdn which can handle the load, then you might not notice for a while i.e. eye popping bill).

Narrowing down the attack profile means looking at logs. Be that network flow data (very helpful) or in this instance web server logs. Probably something like: totals grouped by ip, destination url, etc to see if there are any spikes.

Also, managing stress. If you are some type of retailer then you likely are losing money, people are asking for updates, etc. This can be extremely stressful.

Re:This is what a DDoS attack looks like by etix on Friday April 26, 2013 @01:05PM (#5613999)
We use Munin for the graphs and Nagios for the alerts. The mitigation of the DDoS is done manually since we need to identify the patterns and block them.
Re:This is what a DDoS attack looks like by fsckin on Friday April 26, 2013 @01:12PM (#5614053)
Pretty cool stuff! glTail [0] does similar visual analysis of pretty much anything.

[0] http://www.fudgie.org/

Re:This is what a DDoS attack looks like by fudge on Saturday April 27, 2013 @03:53AM (#5617132)
I currently have a native OSX version og glTail in review by Apple, which is quite a bit faster and easier to install than the Ruby version. I also have an updated IOS version in the queue which doesn't crash quite as often.
Re:This is what a DDoS attack looks like by fsckin on Sunday April 28, 2013 @03:41AM (#5620514)
An IOS version? How cool! I'll check it out, thanks for the heads up.
Re:This is what a DDoS attack looks like by eterm on Friday April 26, 2013 @11:37AM (#5613489)
Fantastic! I had read about such a log visualisation tool a long time ago (I'm not sure but I think I read it about it via NTK which should date it) but I had lost any knowledge of what it might be until now.

Now I can see such a tool and it looks wonderful.

(More on topic, DDOS is beautiful!)

Re:This is what a DDoS attack looks like by ck2 on Saturday April 27, 2013 @05:01AM (#5617227)
Brave of them to disclose it's just the user-agent they are filtering.

It's not possible to inspect the user-agent via the linux firewall (iptables) is it?

I guess you can use this if your iptables supports string matching

    --string "useragent"
Re:This is what a DDoS attack looks like by superflit on Friday April 26, 2013 @10:45PM (#5616603)
I just got one of my servers attacking TicketMaster by a faulty cgi. (my alert system notified 5 minutes after it started) The mob is angry now It was disabled.. I think it has more targets that only vlc...
Re:This is what a DDoS attack looks like by bentaber on Friday April 26, 2013 @12:30PM (#5613771)
Is each request a unique IP or do you see frequent and recurring requests from a chunk of IPs?
Re:This is what a DDoS attack looks like by etix on Friday April 26, 2013 @12:35PM (#5613800)
Depends, I would say half of them are returning every once in a while and others only do a single request and never come back. No clear pattern here.
Re:This is what a DDoS attack looks like by looneylv on Friday April 26, 2013 @02:47PM (#5614688)
I guess someone took some VLC crash seriousley to the heart.... :D

Hang on guys!

Re:This is what a DDoS attack looks like by dfc on Friday April 26, 2013 @01:14PM (#5614077)
I was browsing HN on a friend's computer that without adblock and clicked this link. Wow! Is this what the internet looks like without adblock? The ad/content ratio is crazy...
Re:This is what a DDoS attack looks like by Tobu on Friday April 26, 2013 @07:51PM (#5616170)
Without blogspam: https://youtu.be/hNjdBSoIa8k

Also follow the submitter link, currently etix seems to be most active on Twitter: https://twitter.com/etixxx

Re:This is what a DDoS attack looks like by verandaguy on Friday April 26, 2013 @02:51PM (#5614712)
The VLC logo is a traffic cone.

That visualization sort of looks like a cone.

Coincidence?