News for hackers. Stuff that matters.
Re:Online tracking: A 1-million-site measurement and analysis by beeker87 on Thursday May 19, 2016 @05:36PM (#11734074)
In your paper you say,

"When using the headless configuration, we are able to run up to 10 stateful browser instances on an Amazon EC2 “c4.2xlarge” virtual machine."

Also it seems like you ran the crawl only in the month of January this year, and crawled about 90 million pages. Were you able to do that on the single AWS instance, using Firefox via Selenium? What do you think the performance would have been just issuing raw requests?

Just interested because I'm currently building a crawler and am trying to decide if Selenium would be worth it performance wise.

There are no comments on this comment yet.