Re:Shoestring Budget? Starting to feel growth issues on you...
by corentin on Wednesday September 10, 2008 @02:42AM
(#300173)
> I'm still looking for ways to optimize things
Re-implement your crawler on a FPGA.
Just kidding :)
It's great that you got what you were expecting, but as Paul said, you have to profile before you optimize (otherwise you will "optimize" useless stuff, and not only waste your time but also likely do counter-optimizations).
Anyway, your post was very interesting. Because a lot of people assume they have to use layers on top of the OS, while modern Unix systems have good file systems and memory managers (maybe have a look at DragonFly BSD; they are going into an interesting direction).
Re-implement your crawler on a FPGA.
Just kidding :) It's great that you got what you were expecting, but as Paul said, you have to profile before you optimize (otherwise you will "optimize" useless stuff, and not only waste your time but also likely do counter-optimizations).
Anyway, your post was very interesting. Because a lot of people assume they have to use layers on top of the OS, while modern Unix systems have good file systems and memory managers (maybe have a look at DragonFly BSD; they are going into an interesting direction).