I'm using a linked-list in C to store all of the queued-up URLs so all of the threads can mark them as being worked on or whatever. Also, the app and curl lib takes a few megs too. I'm sure that the heap and multi-threadding libs are responsible for some of the resident memory usage. :) the on-disk binary is only 12k if that make a difference. ;)
The struct:
typedef struct my_url {
char * url;
int retcode;
double old_size;
double new_size;
status status;
struct my_url * next;
char * filename;
int flashurl_id;
int depth;
} my_url;
I also like keeping them in memory as opposed to a BDB because of access speed - 25+ threads don't have to wait when it's all in memory, but 25+ threads hitting a BDB would be much nastier on the OS.
root 6984 3.0 0.5 215832 5168 pts/5 Sl+ 12:59 0:00 ./crawl
I'm using a linked-list in C to store all of the queued-up URLs so all of the threads can mark them as being worked on or whatever. Also, the app and curl lib takes a few megs too. I'm sure that the heap and multi-threadding libs are responsible for some of the resident memory usage. :) the on-disk binary is only 12k if that make a difference. ;)
The struct:
typedef struct my_url { char * url; int retcode; double old_size; double new_size; status status; struct my_url * next; char * filename; int flashurl_id; int depth; } my_url;
I also like keeping them in memory as opposed to a BDB because of access speed - 25+ threads don't have to wait when it's all in memory, but 25+ threads hitting a BDB would be much nastier on the OS.