News for hackers. Stuff that matters.
Re:Posterous is being DDoS'd by rantfoil on Friday August 13, 2010 @03:24PM (#1601963)
We're getting about 500k packets per second, 500mbps to 1.5gbps peak, it's a synflood from a botnet. Typically we can IP hop and null-route the old IP's. That usually buys us about a day until the botnet phones home to get the new IP's, at which point we just hop again. Since our DNS TTL is only 5 minutes at most we are down 5 minutes.

TODAY, the attackers hopped IP's to our new IP immediately. So they appear to be learning. But then again, so are we.

Gigenet's anti-DDoS service has helped us a ton here and is now serving as our front door IP to block the synflood. They've been really responsive.

Re:Posterous is being DDoS'd by axod on Friday August 13, 2010 @07:26PM (#1602592)
Gigenet's service looks interesting. They don't provide pricing though. I'm guessing it's not cheap? Is it based on the bandwidth they'd have to use?
Re:Posterous is being DDoS'd by avar on Friday August 13, 2010 @07:34PM (#1602612)
Relying on a DNS TTL of 5 minutes doesn't work everywhere. Some ISPs will forcefully reset any TTL below 24 hours to 24h.
Re:Posterous is being DDoS'd by thaumaturgy on Friday August 13, 2010 @08:55PM (#1602713)
AT&T/sbcglobal being one.
Re:Posterous is being DDoS'd by sabat on Friday August 13, 2010 @11:39PM (#1602897)
Absolutely true, and completely ridiculous. What, DNS bandwidth is too high? Please.
Re:Posterous is being DDoS'd by avar on Friday August 13, 2010 @11:55PM (#1602914)
Well, not following the RFCs is bad, but I can see why they do it.

Most people using low TTLs probably don't know what they're doing, and if you're a big ISP having to constantly make recursive queries hurts page load times for your customers, who'll blame you.

Re:Posterous is being DDoS'd by wwortiz on Friday August 13, 2010 @06:45PM (#1602521)
I really wish setting DNS TTL's to 5 minutes meant that browsers and everything else (OS caching) would respect that but sadly in my experience it doesn't.

Surprising that it works so well for you.

Re:Posterous is being DDoS'd by weaksauce on Friday August 13, 2010 @04:01PM (#1602082)
Do all the caching name servers respect the TTL though?
Re:Posterous is being DDoS'd by rantfoil on Friday August 13, 2010 @04:03PM (#1602093)
They better. So far so good.
Re:Posterous is being DDoS'd by nroach on Friday August 13, 2010 @03:36PM (#1601999)
Any suggestions for customers with DNS names pointing to posterous blogs? Are we going to need to follow you on each hop?
Re:Posterous is being DDoS'd by rantfoil on Friday August 13, 2010 @03:42PM (#1602020)
Custom domains are unaffected, no action needed.
Re:Posterous is being DDoS'd by catch23 on Friday August 13, 2010 @03:42PM (#1602021)
I wouldn't think so...
Re:Posterous is being DDoS'd by superjared on Friday August 13, 2010 @02:14PM (#1601713)
This is hot off the heels of an email sent out last night:

As you’re no doubt aware, Posterous has had a rocky six days.

On Wednesday and Friday, our servers were hit by massive Denial of Service (DoS) attacks. We responded quickly and got back online within an hour, but it didn’t matter; the site went down and our users couldn’t post.

On Friday night, our team worked around the clock to move to new data centers, better capable of handling the onslaught. It wasn’t easy. Throughout the weekend we were fixing issues, optimizing the site, some things going smoothly, others less so.

Just at the moments we thought the worst was behind us, we’d run up against another challenge. It tested not only our technical abilities, but our stamina, patience, and we lost more than a few hairs in the process.

I’m happy to report Posterous is at 100% and better than ever. Switching to a new data center will help us avoid the type of attacks we saw last week, and the new, bigger, beefier servers will speed up the site and increase capacity. We were hit pretty hard, but we’ve come out stronger in the end.

While we were certainly frustrated, we know that no one was more frustrated than you. Your website was down, and I humbly apologize for that. Know that throughout these six days, restoring your site and your trust has been our number one priority.

At the same time, I have never been prouder of the Posterous team. Garry led the team in the server migration, Vince worked like a mad man until he passed out on his desk, and Jackson helped the community understand the impacts of some pretty technical issues. This week has been all hands on deck, and we had some pretty great hands.

We have great plans for the future of Posterous. Our team is stronger than ever, and we are laser focused on developing better ways for you to share your content online. Thank you for believing in us, thank you for trusting us, and thank you for sticking with us through the past six days.

--Sachin Agarwal Cofounder and CEO, Posterous

Re:Posterous is being DDoS'd by ryanhuff on Friday August 13, 2010 @02:39PM (#1601811)
This sort of crisis management is always a challenge. How do you communicate this sort of problem to customers without losing credibility later if there is a risk of lingering issues? Based on the tone of the email that you posted, as a customer, I would expect the problem solved, case closed. Certainly, it makes customers feel good that the team has it handled.

The new events, if not handled quickly, could create doubt in customer minds and the team, as the optimistic outlook may not hold. I hope they beat this soon.

Re:Posterous is being DDoS'd by zaidf on Friday August 13, 2010 @03:47PM (#1602031)
Simple rule about DDoSes: there are no guarantees.
Re:Posterous is being DDoS'd by vox on Friday August 13, 2010 @02:36PM (#1601802)
Why DDOS posterous? What's there to gain from it?

This isn't like the ones who DDOS'ed MS out of their hatred for it or the ones who blackmailed & DDOS'ed a gambling website when they refused to pay up.

Re:Posterous is being DDoS'd by petercooper on Friday August 13, 2010 @02:40PM (#1601818)
I severely doubt this line of thought.. but they didn't make any friends with their "switch from X to us" campaign, and it only takes one bruised ego with a lack of ethics to stir up trouble.

That said, if it's like most other DDOS attacks I've seen reported, the target is probably a user of the service with Posterous merely being unlucky enough to host a particular site someone doesn't like. I vaguely recall another platform got DDOSed recently due to some Israel vs Iran type cyberwar.

Re:Posterous is being DDoS'd by kiba on Friday August 13, 2010 @02:49PM (#1601844)
Must be something that they really dislike.
Re:Posterous is being DDoS'd by mishmash on Friday August 13, 2010 @03:18PM (#1601942)
Never underestimate the power of lulz when trying to understand a DDoS.
Re:Posterous is being DDoS'd by shaunxcode on Friday August 13, 2010 @02:40PM (#1601814)
someone probably has an issue w/ the blog/content of someone who is using posterous rather than posterous themselves.
Re:Posterous is being DDoS'd by jackowayed on Friday August 13, 2010 @03:35PM (#1601997)
That's my guess as well.

Weebly said (I think in their YC Founders at Work interview) that they get multiple DDoSes a day, aimed not at them but at sites they host. Now they have good enough systems in place to deal with it that they said they often just don't notice.

I guess when you host enough people's content, it's only a matter of time before someone wants to DoS something you're hosting.

Re:Posterous is being DDoS'd by oscardelben on Friday August 13, 2010 @02:41PM (#1601821)
Maybe, and take it with a grain of salt, some people got infuriated by their recent switch to posterous campaign. I know this may be a very remote option, but in my opinion it's very possible.
Re:Posterous is being DDoS'd by lanstein on Friday August 13, 2010 @02:59PM (#1601879)
"A grain of salt" is the expression.
Re:Posterous is being DDoS'd by oscardelben on Friday August 13, 2010 @04:36PM (#1602191)
Thank you
Re:Posterous is being DDoS'd by fragmede on Friday August 13, 2010 @05:28PM (#1602347)
The botnet controller could be extorting websites directly... wire $5000 to my paypal account or your website will go down. Anyone from Posterous care to confirm/deny in this case?
Re:Posterous is being DDoS'd by rdl on Friday August 13, 2010 @10:44PM (#1602848)
This is very common with online gaming (== gambling) sites, and payment systems; anything where there is obviously money being processed.
Re:Posterous is being DDoS'd by devin on Friday August 13, 2010 @06:26PM (#1602479)
Do you have any evidence that this has ever been done in the past?

It's not inconceivable, but it seems like a huge leap.

Re:Posterous is being DDoS'd by orborde on Friday August 13, 2010 @07:41PM (#1602623)
Google seems to provide plenty of examples. Even if it didn't, I don't understand why you see this extortion as a stretch.
Re:Posterous is being DDoS'd by SkyMarshal on Saturday August 14, 2010 @01:41PM (#1603688)
It's not uncommon. There was an article a long time ago telling the story of a DDOS attack on a gambling website, and how a guy who was a philosophy major (iirc) figured out how to beat it, and then formed a company providing the same service. Forgot the url/title/etc, but it was good. Plenty more available with a search.
Re:Posterous is being DDoS'd by DotSauce on Saturday August 14, 2010 @12:26AM (#1602944)
Could this have anything to do with Posterous' risqué post claiming superiority over Tumblr?

http://www.blogherald.com/2010/06/24/posterous-slanders-tumb...

Re:Posterous is being DDoS'd by didip on Friday August 13, 2010 @06:51PM (#1602532)
Who knew what the attackers are thinking. Not too long ago DnsMadeEasy was DDoS from Korea.

Maybe there's financial incentives?

Re:Posterous is being DDoS'd by mrduncan on Friday August 13, 2010 @02:34PM (#1601791)
Our datacenter is experiencing heavy packet loss. We're on the line with Rackspace now.

I don't see where they stated it was a DDOS attack. Packet loss can occur due to a large number of different issues.

EDIT: They just clarified with the following: The DDoS attackers have returned and evolved their attack around our countermeasures. We expect to be back online ASAP w/ @gigenet antiddos

Re:Posterous is being DDoS'd by nroach on Friday August 13, 2010 @03:18PM (#1601945)
It's interesting that they're basically saying Rackspace's network couldn't hold up and doesn't have an effective solution for DDOS victims on their network.

Gige's DDOS protection is basically a hosted redirect and filtering system that "leverage[s] the cost of DDoS mitigation amongst a large group of businesses, giving you access to the infrastructure that would normally be out of reach financially." http://www.gigenet.com/ddos-protection.html

But if Posterous is hosted on the Rackspace cloud, why isn't there a mechanism to do this through their existing host? After all, Rackspace has an equivalent service (maybe rebranded?) that would seem to offer protection without needing to go to a 3rd party vendor. http://www.rackspace.com/managed_hosting/services/security/d...

So it's interesting that Posterous needed to go around Rackspace to get a solution.

Re:Posterous is being DDoS'd by rantfoil on Friday August 13, 2010 @04:02PM (#1602088)
It's tough -- if someone is hitting you at > 1gbps, that exceeds a gigabit ethernet port -- typically on shared services you only pay for that much. Any more than that and it affects all the other customers on the shared network. So yes, Rackspace has the bandwidth for it, but at the same time if you were Rackspace, would you let our DDoS take down the other hosts in the datacenter?

Gigenet / Black Lotus / Prolexic comes in and says -- we have excess bandwidth, we'll take care of it. Then they proxy clean traffic to you. Unfortunately as we are learning, it's also very expensive.

Re:Posterous is being DDoS'd by alphadog on Friday August 13, 2010 @03:51PM (#1602045)
I seriously doubt this is the reason but they insulted a lot of people with their "switch to us" targeted attacks.

That was classless.

Re:Posterous is being DDoS'd by Malcolm_Lloyd on Friday August 13, 2010 @03:59PM (#1602073)
It wasn't classless, it was marketing. No one wants their competitors poaching their clients but in business you deal with it. That said, the same thought went through my mind, I wonder if it's related.
Re:Posterous is being DDoS'd by alphadog on Friday August 13, 2010 @04:12PM (#1602123)
That sounds very clear and professional. Unfortunately, it doesn't always work out that way.

Most of their competitors are in the same situation they are in.

Being the first one to torpedo the other ship and not expecting someone to get upset or God forbid do something awful in return is naive.

I think there is a difference in poaching your competitor's customers and claiming they are "no good".

Re:Posterous is being DDoS'd by paolomaffei on Saturday August 14, 2010 @08:25AM (#1603275)
Can someone estimate the cost of this attack? It shouldn't be cheap to lose all these bots for a massive 1gbps attack.
Re:Posterous is being DDoS'd by OoTheNigerian on Saturday August 14, 2010 @06:25AM (#1603218)
I could claim with nearly all certainty that this is a result of their campaign. The attacked several communities. Not just startups, but communities with developers (wordpress, tumblr etc). Their marketing was bold; but looking back they might have been more subtle.
Re:Posterous is being DDoS'd by gaiusparx on Friday August 13, 2010 @02:31PM (#1601782)
So this is the 2nd DDOS attack. Can DDOS attack be fended off?
Re:Posterous is being DDoS'd by rantfoil on Friday August 13, 2010 @05:23PM (#1602333)
Usually it means using a hardware solution like Intruguard or going through a third party vendor that resells Intruguard and/or other proprietary anti-DDOS technology.

We've opted to use Gigenet, and they're holding the line right now quite well.

Re:Posterous is being DDoS'd by bradgessler on Friday August 13, 2010 @03:44PM (#1602024)
By crushing the enemy.

I hope Posterous does a write-up about this on Axon Flux when this is all over. Poor fellars.

Re:Posterous is being DDoS'd by Mistone on Friday August 13, 2010 @02:51PM (#1601850)
2nd DDOS in a matter of a week.
Re:Posterous is being DDoS'd by paolomaffei on Saturday August 14, 2010 @08:20AM (#1603273)
I wonder if you guys know who's doing it, and why.
Re:Posterous is being DDoS'd by codemechanic on Friday August 13, 2010 @02:18PM (#1601725)
Yes. It is getting DDOS'd now. I just posted a post. But can't access it.