News for hackers. Stuff that matters.
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by afandian on Wednesday July 26, 2017 @05:43PM (#14860642)
I've got the impression that lots of sites block AWS IP addresses. I wonder if this would hamper the practical use of this on Lambda.

I'm doing something similar, and this concern was one motivation for running in our datacentre vs EC2.

Does anyone have concrete info on rates of bots blocked from AWS IPs?

Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by extra88 on Wednesday July 26, 2017 @06:40PM (#14861039)
I assume the number one use of this would be test automation for one's own sites so blocking would not be an issue.

What are sites' motivations for blocking AWS IPs? I bet there are some reasons I would agree with even though the somewhat crude method of blocking ip range would have some unintended consequences (e.g. blocking people running a personal VPN).

Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Wednesday July 26, 2017 @08:24PM (#14861631)
>What are sites' motivations for blocking AWS IPs?

I block AWS. So many crawlers up to so much nonsense! I don't block by IP, but by hostname.

  $block='.amazonaws.com';
  $ua = @$_SERVER['HTTP_USER_AGENT'];
  
  if (stripos($rh,$block)!==false &&
  	stripos($ua,'Silk')===false &&
  	stripos($ua,'Safari')===false){
  
    	$block_visitor=true;
  	$message="Blocked Host:Amazon Web Services";
  }
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by hackits on Thursday July 27, 2017 @12:42AM (#14862768)
Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Thursday July 27, 2017 @04:25AM (#14863542)
Well, from amazonaws.com, there are so many requests for wp-login.php!

And then all the off-brand scraping companies use amazonaws.com.

Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by afandian on Thursday July 27, 2017 @05:34AM (#14863772)
What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Thursday July 27, 2017 @02:07PM (#14867864)
Some AWS visitors:

Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/

That last one is a "SEO platform".

Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by mrmondo on Thursday July 27, 2017 @04:30AM (#14863563)
Indeed, so many of AWS’s IP ranges are used for DDoS and malicious behaviour they end up getting blacklisted due to their poor reputation. It’s a bit like using one third party resellers shared IP ranges in your mail relay - you’re asking to end up on reputation based lists. There’s a lot to be said for your IP reputation on the internet and when you outsource that - you outsource your freedom to maintain your reputation risk.
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by ingenuous2 on Wednesday July 26, 2017 @10:21PM (#14862191)
Fairly simple (if this is like the other programmatic headless browsers) to make a request through a proxy.
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by afandian on Thursday July 27, 2017 @05:42AM (#14863809)
And where do you run the proxy?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by bdcravens on Wednesday July 26, 2017 @11:21PM (#14862444)
In my experience this isn't happening.