Re:Scraping Roger Ebert’s reviews and finding his favorite m...
by catwind7 on Wednesday June 10, 2020 @01:13PM
(#23479536)
I actually did notice that issue, even with using a stateful client like mechanize. Sometimes I had to scrape > 5 times in order to get through the "anti robot" page.
Other times, I get no issue at all. It's weird - maybe they're doing some pattern matching on request metadata on their end?
Other times, I get no issue at all. It's weird - maybe they're doing some pattern matching on request metadata on their end?