Re:Ask HN: Is there a great open source crawler?
by haihai on Thursday June 03, 2010 @12:57PM
(#1401818)
I considered 80legs. The problem is that I don't want to do this for just one site. I may have dozens. I can run this off my own servers/bandwidth (which I already pay for) for much cheaper than using 80legs.
If 80legs was purely usage based and cheaper I probably would have used it.
Re:Ask HN: Is there a great open source crawler?
by jdrock on Thursday June 03, 2010 @01:39PM
(#1401932)
Understood. We do work with folks on custom per-use plans, but we just need to figure out what your requirements are so that we can see if there's some way for us to help.
If 80legs was purely usage based and cheaper I probably would have used it.