Re:Google's Indexing Javascript more than we thought
by paganel on Friday December 02, 2011 @05:20PM
(#3305970)
> I am not saying all the bots are smart, but my mantra since then has been that "if there is value for the bots to be smart, they can get very smart".
I was once actually on the other side of the fence as you were, around 5-6 years ago (in a different industry, though). You're right, if there's value to be gained by scrapping other people's pages there's almost always a way round the obstacles.
I remember the day when my boss presented me a link to a strange-named FF extension, called Chickenfoot (http://groups.csail.mit.edu/uid/chickenfoot/faq.html). It allowed one to very easily write FF extensions that would programatically click on whatever links you wanted to be scrapped, all this from inside the browser, like a normal user would have done. I used to run FF with this extension installed on a dedicated cheap PC, saving the data to our servers, and from time to time automatically restarting FF because the machine was running out of memory. Fun times :)
I was once actually on the other side of the fence as you were, around 5-6 years ago (in a different industry, though). You're right, if there's value to be gained by scrapping other people's pages there's almost always a way round the obstacles.
I remember the day when my boss presented me a link to a strange-named FF extension, called Chickenfoot (http://groups.csail.mit.edu/uid/chickenfoot/faq.html). It allowed one to very easily write FF extensions that would programatically click on whatever links you wanted to be scrapped, all this from inside the browser, like a normal user would have done. I used to run FF with this extension installed on a dedicated cheap PC, saving the data to our servers, and from time to time automatically restarting FF because the machine was running out of memory. Fun times :)