News for hackers. Stuff that matters.

A comment on Tiny, dirty, iffy, good enough, basic multi-threaded web crawler in Python. Parent comment

Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by tokenizerrr on Wednesday August 12, 2015 @09:23AM (#10047542)
If you're crawling different origins in parallel then you could gain a boost in speed by threading. It's usually bound by the network/processing on the receiving side. Your server can probably handle several outgoing connections at once.
[ Parent | on HN ]
Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by sebcat on Wednesday August 12, 2015 @12:22PM (#10048674)
You can have several outgoing connections in one single thread, at the same time.

Threading is for CPU bound problems that requires a shared memory space.

I/O multiplexing and/or cooperative multitasking means you can have thousands concurrent connections per thread.

[ Parent | on HN ]
Re:Tiny, dirty, iffy, good enough, basic multi-threaded web ... by tokenizerrr on Wednesday August 12, 2015 @03:06PM (#10049917)
Sure. The parent comment referred to one download per thread so I did as well.
[ Parent | on HN ]