Re:Some analysis of the 1M most popular sites on the web
by heyalexej on Friday July 24, 2015 @12:55PM
(#9943359)
This is very interesting. Will you release the data and code at some point?
[ Parent
| on HN ]
Re:Some analysis of the 1M most popular sites on the web
by jacquesm on Friday July 24, 2015 @01:01PM
(#9943409)
Yes, I will definitely release the code and the dataset required to bootstrap the rest. It takes a long long time to run and you'll need a good bit of bandwidth. I won't be releasing the raw data because there is simply too much of it.
Re:Some analysis of the 1M most popular sites on the web
by adriancooney on Friday July 24, 2015 @01:22PM
(#9943584)
If bandwidth is a worry there are open data projects by Amazon[1] and I think Google that could host your data.
[1] Amazon Public Data Sets, http://aws.amazon.com/public-data-sets/
[2] Google Public Data, http://www.google.com/publicdata/directory
Re:Some analysis of the 1M most popular sites on the web
by heyalexej on Friday July 24, 2015 @02:22PM
(#9944104)
Sorry for bugging you. Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content?
I crawl large-ish websites (most recently https://code.google.com with 1.8MM repos) often and am really looking forward to your dataset & code.
[1] http://phantomjs.org/api/webpage/handler/on-resource-receive...
Re:Some analysis of the 1M most popular sites on the web
by jacquesm on Friday July 24, 2015 @02:36PM
(#9944198)
> Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content?
That would have constrained throughput too much so I opted for culling it during the crawl to just content-type and url, this was then processed to extract the various bits of information. I did use the 'resource received' trick you linked above. Very useful.