Re:Measuring the impact of Google Analytics (2013)
by valjavec on Monday May 12, 2014 @09:23AM
(#7732507)
How much was cost for all that? Or how much would it cost to process 1m pages like this. Do some entity extraction on each page?
[ Parent
| on HN ]
Re:Measuring the impact of Google Analytics (2013)
by Encosia on Monday May 12, 2014 @11:55AM
(#7733464)
A few years ago, I wrote a little crawler to run through the top 200k sites on Alexa, search for script references, and log them to a database, to get a sense for what the real usage of Google's jQuery CDN was in the wild[0]. IIRC, that took less than a day to run on the consumer broadband I was using at the time.
[0]: http://encosia.com/6953-reasons-why-i-still-let-google-host-...
Re:Measuring the impact of Google Analytics (2013)
by Smerity on Monday May 12, 2014 @12:01PM
(#7733508)
For only one million web pages, the job would likely be quite cheap. The Common Crawl corpus is hundreds of millions of pages and, given the right setup, only takes $10 to $100 to process, especially for relatively light entity extraction. More expensive operations, such as parsing using NLP tools, will obviously be more expensive.