Re:An exploration of Yelp's own filtered reviews
by SoftwareMaven on Sunday September 04, 2011 @07:38PM
(#2960584)
That's what robots.txt is for, not silly captchas. You put captchas for the people who ignore that file.
Re:An exploration of Yelp's own filtered reviews
by derwiki on Sunday September 04, 2011 @08:37PM
(#2960693)
I think the idea is that given a large corpus of filtered and unfiltered reviews, you might be able to reverse engineer signals in the algorithm and game the system. If that's your end goal, you and the software you write is likely to ignore robots.txt directives.
Re:An exploration of Yelp's own filtered reviews
by pud on Sunday September 04, 2011 @07:57PM
(#2960615)
Not all spiders honor robots.txt
Re:An exploration of Yelp's own filtered reviews
by progolferyo on Sunday September 04, 2011 @07:41PM
(#2960592)
Plus captcha's are just such a stupid user experience anyway, if you want to avoid the robots problem, there are plenty of ways around captchas
Re:An exploration of Yelp's own filtered reviews
by jonknee on Sunday September 04, 2011 @08:08PM
(#2960638)
Like? Yelp doesn't want the filtered reviews to be accessed in an automated fashion, that's what a CAPTCHA does. What are the other options?
Re:An exploration of Yelp's own filtered reviews
by progolferyo on Sunday September 04, 2011 @08:16PM
(#2960654)
A quick and dirty solution could be to add something to the page using javascript after the page has loaded and only let the link work if that variable exists (and check the value of the key with the server, if you wanted to be more cautious). Not a complete solution, but a first step and invisible to the user (and a pain in the ass to a robot)
Re:An exploration of Yelp's own filtered reviews
by waitwhat on Sunday September 04, 2011 @08:53PM
(#2960733)
A robot scraping yelp's deep data is going to be site-specific, and having to scrape another javascript variable is not much more than a slight speed-bump.
Maybe they don't want Google to see them...