News for hackers. Stuff that matters.
Re:Crawling Billions of Pages: Building Large Scale Crawling... by rb2k_ on Thursday June 11, 2015 @10:17PM (#9703649)
I guess this fits in here:

Once upon a time I wrote my thesis on building a web crawler. The (tiny) blog post with an embedded preview:

http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo...

The PDF itself:

http://blog.marc-seeger.de/assets/papers/thesis_seeger-build...

It's mostly a "this is what I learned and the things I had to take into consideration" with a few "this is how you identify a CMS" bits sprinkled into it. These days I would probably change a thing or two, but people told me it's still an entertaining read. (Not a native speaker though, so the English might have some stylistic kinks)

There are no comments on this comment yet.