News for hackers. Stuff that matters.
Re:Detecting Chrome headless by Dolores12 on Saturday August 05, 2017 @01:45PM (#14936869)
Every spider process is a browser tab. People have hundreds tabs opened without any problem on ordinal computers. On a dedicated server with 128GB of ram you can run thousands of those and it will cost you 100$-200$/month.
Re:Detecting Chrome headless by jasonjayr on Saturday August 05, 2017 @05:01PM (#14937872)
True -- But the user is not interacting with all 100's of those tabs at the same time -- something an automated scraper would be.

Firefox + Chrome both lower the priority of background tabs, and may be doing other tricks so the background tabs can stick around and be switched to quickly.

Re:Detecting Chrome headless by Dolores12 on Sunday August 06, 2017 @02:58AM (#14940131)
Well, most of the time spiders are idle while waiting for IO operations to complete. Main bottleneck is not CPU, but RAM and bandwidth.
Re:Detecting Chrome headless by rileymat2 on Sunday August 06, 2017 @09:31AM (#14941225)
It depends on what analysis is being done on each page.
Re:Detecting Chrome headless by Dolores12 on Sunday August 06, 2017 @11:05AM (#14941580)
What kind of analysis are you expecting from a crawler which main purpose is to grab a webpage?
Re:Detecting Chrome headless by ivanhoe on Sunday August 06, 2017 @04:55PM (#14943479)
Main purpose is to render the page, crawl the dom for the data, and then load the next page as fast as possible... so it's equivalent of having a thousand bookmarks and opening them all at the same time, and as soon each loads running the scraper script and reloading them with the new link. Try it for yourself, it's easy to script, and check the load...
Re:Detecting Chrome headless by rileymat2 on Sunday August 06, 2017 @06:29PM (#14944045)
Presumably, there is some reason you are crawling.
Re:Detecting Chrome headless by MuffinFlavored on Monday August 07, 2017 @09:48AM (#14947378)
Where can I get a dedicated server with 128GB of RAM for $100-$200/mo?