News for hackers. Stuff that matters.
Re:Some analysis of the 1M most popular sites on the web by jacquesm on Friday July 24, 2015 @02:29PM (#9944150)
> How deep did you crawl?

Front pages only.

> I would have guessed the flash usage to be higher.

When adding all the pages in a site it no doubt will be. I'll update the article to clarify this.

> How big is the dataset?

In flight: huge, but after culling and keeping only the bits that I needed it was a lot smaller, about 20G.

> How long did it take?

About 10 days.

> Which tools did you use besides phantomjs?

Just some php glue scripts, nothing fancy, about 500 lines.

There are no comments on this comment yet.