News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by no_time on Monday September 13, 2021 @03:24PM (#28515629)
While a decent post, this is more or less inadaquate in 2021. Do a post on bypassing cloudflare/other anti botting tech using residental proxy swarms
Re:How to Crawl the Web with Scrapy by mmerlin on Monday September 13, 2021 @04:13PM (#28516234)
Yes, Scrapy is quite a good scraper technology for some features, especially caching, but for some websites it's like doing things the hard way...

The easiest scraper with a proxy rotator I've found is in my current fave web-automator, scraper scripter and scheduler: Rtila [1]

Created by an indy/solo developer-on-fire cranking out user-requested features quite quickly... check the releases page [2]

I have used (or at least trialled) the vast majority of scraper-tech and written hundreds of scrapers since my first VB5 controlling IE then dumping to SQLserver in the 90's and then moving to various php and python libs/frameworks and a handful of windows apps like ubot and imacros (both of which were useful to me at some point but I never use those nowadays)

A recent release of Rtila allows creating standalone bots you can run using it's built-in local Node.js server (which also has it's own locally hosted server API you can program anything else against using any language you like)

[1] https://www.rtila.net

[2] https://github.com/IKAJIAN/rtila-releases/releases

Re:How to Crawl the Web with Scrapy by Lammy on Monday September 13, 2021 @05:01PM (#28516827)
I'm sure Rtila is fantastic at what it does, but I gotta say it's hilarious to see a landing page done in the Corporate Memphis artstyle but worded in euphemism: https://www.rtila.net/#h.d30as4n2092u

"‘Cause if the web server said no, then the answer obviously is no. The thing is that it’s not gonna say no—it’d never say no, because of the innovation."

Re:How to Crawl the Web with Scrapy by trashcan on Monday September 13, 2021 @10:09PM (#28519376)
That is a modified quote from Its Always Sunny In Philadelphia BTW, if you didn't recognize it.
Re:How to Crawl the Web with Scrapy by matheusmoreira on Monday September 13, 2021 @03:51PM (#28515936)
Yeah. I hate cloudflare, captchas. Why can't these companies accept that our scrapers are valid user agents? Only Google is allowed to do it, nobody else.
Re:How to Crawl the Web with Scrapy by Eikon on Monday September 13, 2021 @04:15PM (#28516267)
Because most scrapers aren't providing any value to website owners, in fact, they are costing them, unlike google.
Re:How to Crawl the Web with Scrapy by r_singh on Monday September 13, 2021 @03:33PM (#28515716)
Exactly!

While there are scraping APIs that unblock requests and charge for them, I'd love to learn more about how they work....

Re:How to Crawl the Web with Scrapy by Eikon on Monday September 13, 2021 @04:19PM (#28516314)
They use residential proxies with altered clients and / or headless browsers. Cloudflare's bot protection mostly makes use of TLS fingerprinting, and thus pretty easy to bypass.
Re:How to Crawl the Web with Scrapy by wswope on Monday September 13, 2021 @04:26PM (#28516417)
Scraping is a cat and mouse game that’ll vary a lot by site. I’m far from an expert and welcome correction here, but the two big tricks that’ll go a long way AFAIK are using a residential proxy service (never tried one - they tend to be quite shady), and using a webdriver-type setup like Selenium or Puppeteer to mock realistic behavior (though IIRC you have to obfuscate both those systems since they’re detectable via JS).