News for hackers. Stuff that matters.
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Wednesday July 26, 2017 @08:24PM (#14861631)
>What are sites' motivations for blocking AWS IPs?

I block AWS. So many crawlers up to so much nonsense! I don't block by IP, but by hostname.

  $block='.amazonaws.com';
  $ua = @$_SERVER['HTTP_USER_AGENT'];
  
  if (stripos($rh,$block)!==false &&
  	stripos($ua,'Silk')===false &&
  	stripos($ua,'Safari')===false){
  
    	$block_visitor=true;
  	$message="Blocked Host:Amazon Web Services";
  }
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by hackits on Thursday July 27, 2017 @12:42AM (#14862768)
Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Thursday July 27, 2017 @04:25AM (#14863542)
Well, from amazonaws.com, there are so many requests for wp-login.php!

And then all the off-brand scraping companies use amazonaws.com.

Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by afandian on Thursday July 27, 2017 @05:34AM (#14863772)
What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Thursday July 27, 2017 @02:07PM (#14867864)
Some AWS visitors:

Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/

That last one is a "SEO platform".