Re:Some analysis of the 1M most popular sites on the web
by jacquesm on Friday July 24, 2015 @02:36PM
(#9944198)
> Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content?
That would have constrained throughput too much so I opted for culling it during the crawl to just content-type and url, this was then processed to extract the various bits of information. I did use the 'resource received' trick you linked above. Very useful.
That would have constrained throughput too much so I opted for culling it during the crawl to just content-type and url, this was then processed to extract the various bits of information. I did use the 'resource received' trick you linked above. Very useful.