News for hackers. Stuff that matters.

A comment on Show HN: Chromeless – Headless Chrome Automation on AWS Lambda. Parent comment

Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by hackits on Thursday July 27, 2017 @12:42AM (#14862768)
Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
[ Parent | on HN ]
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Thursday July 27, 2017 @04:25AM (#14863542)
Well, from amazonaws.com, there are so many requests for wp-login.php!

And then all the off-brand scraping companies use amazonaws.com.

[ Parent | on HN ]
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by afandian on Thursday July 27, 2017 @05:34AM (#14863772)
What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
[ Parent | on HN ]
Re:Show HN: Chromeless – Headless Chrome Automation on AWS L... by pravda on Thursday July 27, 2017 @02:07PM (#14867864)
Some AWS visitors:

Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/

That last one is a "SEO platform".

[ Parent | on HN ]