I think the demands on concurrency are relatively basic. It's nice to have something a bit more robust than raw pthreads, but the main thing that makes or breaks a crawler is access to a robust HTTP client library and HTML parser, solid I/O performance, that sort of thing. Because the problem domain looks the way it looks, concurrency is one of the easier parts.