Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by extra88 on Wednesday July 26, 2017 @06:40PM
(#14861039)
I assume the number one use of this would be test automation for one's own sites so blocking would not be an issue.
What are sites' motivations for blocking AWS IPs? I bet there are some reasons I would agree with even though the somewhat crude method of blocking ip range would have some unintended consequences (e.g. blocking people running a personal VPN).
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by pravda on Wednesday July 26, 2017 @08:24PM
(#14861631)
>What are sites' motivations for blocking AWS IPs?
I block AWS. So many crawlers up to so much nonsense! I don't block by IP, but by hostname.
$block='.amazonaws.com';
$ua = @$_SERVER['HTTP_USER_AGENT'];
if (stripos($rh,$block)!==false &&
stripos($ua,'Silk')===false &&
stripos($ua,'Safari')===false){
$block_visitor=true;
$message="Blocked Host:Amazon Web Services";
}
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by hackits on Thursday July 27, 2017 @12:42AM
(#14862768)
Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by pravda on Thursday July 27, 2017 @04:25AM
(#14863542)
Well, from amazonaws.com, there are so many requests for wp-login.php!
And then all the off-brand scraping companies use amazonaws.com.
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by afandian on Thursday July 27, 2017 @05:34AM
(#14863772)
What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by pravda on Thursday July 27, 2017 @02:07PM
(#14867864)
Some AWS visitors:
Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/
That last one is a "SEO platform".
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by mrmondo on Thursday July 27, 2017 @04:30AM
(#14863563)
Indeed, so many of AWS’s IP ranges are used for DDoS and malicious behaviour they end up getting blacklisted due to their poor reputation. It’s a bit like using one third party resellers shared IP ranges in your mail relay - you’re asking to end up on reputation based lists. There’s a lot to be said for your IP reputation on the internet and when you outsource that - you outsource your freedom to maintain your reputation risk.
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L...
by ingenuous2 on Wednesday July 26, 2017 @10:21PM
(#14862191)
Fairly simple (if this is like the other programmatic headless browsers) to make a request through a proxy.
I'm doing something similar, and this concern was one motivation for running in our datacentre vs EC2.
Does anyone have concrete info on rates of bots blocked from AWS IPs?