Re:Ask HN: Is there a great open source crawler?
by jdrock on Thursday June 03, 2010 @01:09AM
(#1400449)
Tooting my own horn, but this takes all of 1 minute in 80legs if you've got the right regex.
[ Parent
| on HN ]
Re:Ask HN: Is there a great open source crawler?
by haihai on Thursday June 03, 2010 @12:57PM
(#1401818)
I considered 80legs. The problem is that I don't want to do this for just one site. I may have dozens. I can run this off my own servers/bandwidth (which I already pay for) for much cheaper than using 80legs.
If 80legs was purely usage based and cheaper I probably would have used it.