Re:How to write a crawler
by rzendacott on Sunday September 29, 2013 @05:01PM
(#6466504)
Udacity CS101 [1] also goes through the basics of building a web crawler. It's a lot more lightweight (no backend, etc), but it's a fun overview and can be completed pretty quickly.
[1]: https://www.udacity.com/course/cs101