News for hackers. Stuff that matters.
Re:Measuring the impact of Google Analytics (2013) by valjavec on Monday May 12, 2014 @09:23AM (#7732507)
How much was cost for all that? Or how much would it cost to process 1m pages like this. Do some entity extraction on each page?
Re:Measuring the impact of Google Analytics (2013) by Encosia on Monday May 12, 2014 @11:55AM (#7733464)
A few years ago, I wrote a little crawler to run through the top 200k sites on Alexa, search for script references, and log them to a database, to get a sense for what the real usage of Google's jQuery CDN was in the wild[0]. IIRC, that took less than a day to run on the consumer broadband I was using at the time.

[0]: http://encosia.com/6953-reasons-why-i-still-let-google-host-...

Re:Measuring the impact of Google Analytics (2013) by Smerity on Monday May 12, 2014 @12:01PM (#7733508)
For only one million web pages, the job would likely be quite cheap. The Common Crawl corpus is hundreds of millions of pages and, given the right setup, only takes $10 to $100 to process, especially for relatively light entity extraction. More expensive operations, such as parsing using NLP tools, will obviously be more expensive.