Re:How to crawl a quarter billion webpages in 40 hours (2012)
by londons_explore on Thursday June 15, 2023 @07:04AM
(#36338755)
Headless browsers are expensive, but if you needed them, I suspect there are easy wins to be had on the performance front.
For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.
For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.