Sometimes the API will need some form of session tokens found in HTTP meta tags, form inputs, or cookies. Sometimes they have some CORS checks in place that are very easy to bypass from a server-side environment by spoofing the Origin header.
For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.
I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.
If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the content of the page. And they execute the click event handlers on every link/button and when we use "history.pushState()" to change the URL Google considers that a new page.
You're just going to get a loading spinner with no content if you do a dumb crawl (I disagree with that and think we should be running a headless browser server side to execute javascript and generate the initial page content for all our pages... but so far management hasn't prioritised that change... instead they just keep telling us to make our client side javascript run faster... imagine if there was no javascript to execute at all? At least none before first contentfull paint)
This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which.
I don't personally attempt to crawl SPAs because it's not the sort of content I want to index.
That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Content can be user submitted in-browser. Content can be “engaged with” in browser. Some handle this blurring better than others. HN is an example IMO of getting it right where nearly everything that should be addressable (like comments) can be linked to. Others not so much.
(As an aside, I love marginalia!)
It's simply not feasible to treat them as any other website if you wanna index their data.
- The host has a tendency to block unknown user agents, or user agents that claim to be a browser but are not
- Anything that requires client side rendering
I'd suppose both problems are more pertinent in 2023 than they were in 2012.
At web scale, the issue appears on at what point would you be required to use a headless browser when not using one in the first place e.g. if React is included/referenced. Perhaps some simple fingerprinting of JS files would do, IMO in reality the line is very blurry, so you either do or don't.
[1] https://www.adriancooney.ie/blog/web-scraping-via-javascript...
(Previous comments about it from 2022 here: https://news.ycombinator.com/item?id=31205139)
For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need.
For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.
Instead of setting up some kind of partnership with our vendors, where they just send us information or provide an API, we scrape their websites.
The old version ran in a hour, using one thread on one machine. Downloaded PDF's and extracted the values.
The new version is Selenium based, uses 20 cores, 300GB of memory, and takes all night to run. It does the same thing, but from inside of a browser.
As a bonus, the 'web scrapers' were blamed for every performance issue we had for a long time.
How are they scaled?
If you need cookies/headers, you can always open the browser, log in and then make the same requests in the console, instead of waiting for the browser to load and scrape the UI (by xpath, etc.)
sounds weird going in circles: - SPA call some URL - SPA use the response data to populate the UI - You scrape the UI
instead of just calling the url inside the browser? am i missing something?
You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this:
- SPAs with front end only rendering are hard to scrape with a traditional http library.
- Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots.
Using a "real browser" is often advised as a way around these issues.
However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found:
- Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.
- Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want.
- Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.