I thought that most of these solutions deal with CPU-bound and sometimes RAM-bound situations -- i.e. they fix spending too much time spinning the CPU in various ways waiting for I/O, or too many threads at once taking up too much RAM.
The general method appears to be, write the program using an event model, then create a thread per CPU, which each thread waiting on epoll() (timeouts optional).
From what I understand (and how I've used it in my work), one uses epoll() specifically because they're NOT I/O-bound and so need to come up with strategies for using the minimum amount of CPU and RAM per simultaneous I/O so as to avoid becoming CPU- or RAM- bound.
Hence my original point -- if one is I/O-bound while using poll(), it doesn't really matter whether the CPU is spinning on that or epoll(), since I/O won't happen any faster.
Alternative arrangement is to have a single IO thread that deals exclusively with getting data off the wire and multiple "worker" threads that handle the actual processing. I like this model better, because it provides cleaner logic separation, run-time threading control and it has several other advantages. Though "your experience may vary".
If your crawler is I/O-bound, then just wait till you discover epoll :)
Or, on a more general note, have a look at http://www.kegel.com/c10k.html