News for hackers. Stuff that matters.
Re:How to write a crawler by riffraff on Sunday September 29, 2013 @05:49PM (#6466667)
in a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk.

At some point querying the db to check what URLs you have can become quite heavy

There are no comments on this comment yet.