Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Some analysis of the 1M most popular sites on the web. Parent comment
Front pages only.
> I would have guessed the flash usage to be higher.
When adding all the pages in a site it no doubt will be. I'll update the article to clarify this.
> How big is the dataset?
In flight: huge, but after culling and keeping only the bits that I needed it was a lot smaller, about 20G.
> How long did it take?
About 10 days.
> Which tools did you use besides phantomjs?
Just some php glue scripts, nothing fancy, about 500 lines.
There are no comments on this comment yet.
Front pages only.
> I would have guessed the flash usage to be higher.
When adding all the pages in a site it no doubt will be. I'll update the article to clarify this.
> How big is the dataset?
In flight: huge, but after culling and keeping only the bits that I needed it was a lot smaller, about 20G.
> How long did it take?
About 10 days.
> Which tools did you use besides phantomjs?
Just some php glue scripts, nothing fancy, about 500 lines.