Not necessarily. Let's think of some operations a crawler would need to do:

1. Spawn some threads (Python uses native threading) 2. Connect to and load some URLs (This isn't going to be slow anywhere.) 3. Run some regular expressions (Python's regexp engine is all C) 4. Write to the disk. (Python uses the standard system libraries for this)

It wouldn't be as fast, but it might be surprisingly close. There would be fewer lines of code to boot.

The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. It sounds like the man only has 1 processor, so that wouldn't be the end of the world, but if he has more, you can just swap in the process module for the threading module. Of course, then you might have some RAM issues.

Or, briefly: Certain classes of Python programs can actually be faster than average C, because they consist of small amounts of Python code connecting libraries that are already written in very thoroughly tuned C.

Why use threads when you can use an event-based IO system? (I think Twisted is the Python way of doing this.)

I was just going off of what he had already done. I agree that async is a better way to do this.

Twisted is one way, but Python ships with asynchronous libraries http://docs.python.org/lib/module-asyncore.html.

> The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time.

Not actually true; socket calls release the GIL when they're not active, as do many of python's C libraries. Especially the better-written standard ones.

True, although I would argue that my statement is still correct. You cannot have more than 1 thread running simultaneously.

The GIL is not as big of a problem as most people see it, it only interferes with CPU bound, highly parallelizable tasks.

The "..." above had some important bits in it. My point was, how much of the speedup came from switching to C, and how much came from all the other stuff he did (much of which could also be done in Python).

Switching from a scripting language to C doesn't mean an automatic huge speed boost. It depends where your time is being spent.

"everything [he] described" regarding code was porting code over to C.