News for hackers. Stuff that matters.
Re:AWS drops bandwidth pricing by baltcode on Thursday June 30, 2011 @12:00AM (#2712488)
I thought most AWS IP addresses were blocked by most admins. Does any one have experience scraping using Amazon.
Re:AWS drops bandwidth pricing by mike-cardwell on Thursday June 30, 2011 @05:04AM (#2713196)
Most admins? Most admins probably don't even know what AWS is. Most admins probably don't even block a single IP. I would be surprised if as many as 0.1% of admins block AWS.
Re:AWS drops bandwidth pricing by ben1040 on Thursday June 30, 2011 @01:40AM (#2712796)
I don't scrape, but I use an EC2 micro instance as a VPN endpoint when I am on an unsecured wifi network.

It's a little annoying that some sites block me solely because I'm coming in from AWS (StackOverflow, I'm looking at you).

Re:AWS drops bandwidth pricing by koko775 on Thursday June 30, 2011 @05:22AM (#2713231)
I suggest using prgmr.com instead. $6/mo is worth a Xen instance with IPv6 and everything.
Re:AWS drops bandwidth pricing by abronte on Thursday June 30, 2011 @12:13AM (#2712527)
We have a pretty big e-commerce scraping project, and we don't really run into many problems regarding blacklisted ip's. There are a few sites who we get consistent bans from, but with elastic ip's it's pretty much a non-issue. I have yet to see a site that ban's all amazon ip's.
Re:AWS drops bandwidth pricing by sjwright on Thursday June 30, 2011 @12:44AM (#2712642)
I run http://whirlpool.net.au and I religiously check the Amazon EC2 forum announcements[1] for new IP ranges to ban.

[1] https://forums.aws.amazon.com/ann.jspa?annID=1030

Re:AWS drops bandwidth pricing by stoney on Thursday June 30, 2011 @12:54AM (#2712675)
> I run http://whirlpool.net.au

An excellent and very useful forum! It seems like whatever topic I'm searching for, google(.com.au) returns a useful result on your site.

Re:AWS drops bandwidth pricing by aw3c2 on Thursday June 30, 2011 @02:33AM (#2712907)
Would you mind telling why?
Re:AWS drops bandwidth pricing by maaku on Thursday June 30, 2011 @03:49AM (#2713058)
Why?

Name me one good reason. Name me one.

Re:AWS drops bandwidth pricing by sjwright on Thursday June 30, 2011 @03:57AM (#2713066)
Shitloads of rogue bots doing "social media monitoring".

Shitloads of rogue bots stealing content for black-hat SEO.

Shitloads of rogue bots harvesting email addresses.

Shitloads of rogue bots submitting spammy replies.

Re:AWS drops bandwidth pricing by maaku on Thursday June 30, 2011 @12:42PM (#2714735)
So maintain a blacklist of elastic IPs. If it's too big for you, make it a community effort.

Those are bad reasons to close your site to all of AWS.

As nupark2 mentioned, there are legitimate users routing traffic through EC2, even some bots that you'd want to visit your site. Archive.org comes to mind (many of there scrapers are or were behind AWS). Closing your site or app to a large swath of the web is the wrong solution. It's like killing a spider with a bazooka.

Re:AWS drops bandwidth pricing by sjwright on Friday July 01, 2011 @03:01AM (#2717325)
Unlike the assumptions you're limited to making, I know how much of my AWS traffic is human, and it's really very very very small. The sad reality is I'm sick and tired of rogue bots, and the tiny sliver of collateral damage can fill out the CAPTCHA validation every so often.

(I also blacklist GWS, rackspace, linode, softlayer, reliablehosting, ovh.net, node4, netdirect, layer42, all TOR exits... it's actually a pretty huge list.)

Re:AWS drops bandwidth pricing by sjwright on Friday July 01, 2011 @03:03AM (#2717333)
I whitelist archive.org, and they've never hit through AWS.
Re:AWS drops bandwidth pricing by nupark2 on Thursday June 30, 2011 @07:32AM (#2713559)
That's unfortunate. I don't know what the answer is, but real people do route their traffic out of AWS endpoints.
Re:AWS drops bandwidth pricing by sjwright on Friday July 01, 2011 @03:02AM (#2717329)
Far fewer than you might think.
Re:AWS drops bandwidth pricing by mattgreenrocks on Thursday June 30, 2011 @12:22AM (#2712552)
It is annoying when it happens and you're on something like Heroku.
Re:AWS drops bandwidth pricing by semanticist on Thursday June 30, 2011 @06:17PM (#2715953)
In my experience no UK newspapers block AWS IP addresses.
Re:AWS drops bandwidth pricing by Rickasaurus on Friday July 01, 2011 @02:07AM (#2717217)
I usually use Tor when I scrape anyway :)