Tiny, dirty, iffy, good enough, basic multi-threaded web crawler in Python

Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by sebcat on Wednesday August 12, 2015 @09:04AM (#10047462)

Crawling can be broken down to:

  1) fetching resources
  2) finding out what new resources to fetch

1) is an network bound problem, 2) is mostly disk/CPU bound. Realizing the difference between these two things and separating them is the key to building a good crawler.

Depending on how you find out what resources to fetch (parsing static documents vs. dynamic JS analysis with multiple dependencies on other resources (included JS &c)), "good-enough" crawlers are mostly bound to the network.

I've seen people running 1 crawl/process on their back-end and some management guy saying "we need to crawl faster, add more threads per crawl" when one crawl cycle spends 10x times more waiting on the network than it does parsing a document.

[ Parent | on HN ]

Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by tokenizerrr on Wednesday August 12, 2015 @09:23AM (#10047542)

If you're crawling different origins in parallel then you could gain a boost in speed by threading. It's usually bound by the network/processing on the receiving side. Your server can probably handle several outgoing connections at once.

[ Parent | on HN ]

Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by sebcat on Wednesday August 12, 2015 @12:22PM (#10048674)

You can have several outgoing connections in one single thread, at the same time.

Threading is for CPU bound problems that requires a shared memory space.

I/O multiplexing and/or cooperative multitasking means you can have thousands concurrent connections per thread.

[ Parent | on HN ]

Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by tokenizerrr on Wednesday August 12, 2015 @03:06PM (#10049917)

Sure. The parent comment referred to one download per thread so I did as well.

[ Parent | on HN ]