News for hackers. Stuff that matters.
Re:Using PhantomJS at scale by onethumb on Tuesday December 17, 2013 @04:53PM (#6924211)
This is a particularly big issue for us because we spend more than 50% of our resources on serving robots (not humans!).

I wrote about this back in July of 2010, and the problem has only gotten worse: http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh...

It's a crazy problem to have, but we haven't found a solution. We can't just disappear from all the various indexes and collections that these robots represent.

Re:Using PhantomJS at scale by iLoch on Tuesday December 17, 2013 @08:08PM (#6925177)
I face a similar issue going forward. I think http://prerender.io has been posted to HN before - seems to be a pretty reasonable plug and play solution.
Re:Using PhantomJS at scale by rdoherty on Wednesday December 18, 2013 @12:45PM (#6929079)
Thanks for the link! We started working on this project in early 2013, prerender.io didn't exist then (I think).

prerender.io does do most of what we do. A few things we do differently are keeping a pool of PhantomJS workers always available, killing HTTP requests that never complete, putting it behind HAproxy and logging.