Yea, fetching one URL at a time was a major issue, but things like the executable size in memory was also a problem. Remember, this is an old machine. Mysql takes 400M for its index for my data, my other processes run on the same machine also and all take memory. I initially added swap for things, but that's clearly not the right direction.

Also, I'm 40 years old. When I went to college, C programming was the norm, so it wasn't that difficult for me and didn't take long to implement and seemed like a great place to start to optimize things. It's fast, saturates my home Comcast pipe (they might cap my bandwidth, which I'm now worried about) and takes about 200M of memory with 30k URLs loaded into the queue. I suppose that I could reduce the memory requirements of the app some more by keeping the URLs in a BerkDB or something, but I think that 200m is acceptable for my needs now. For me, it's all about getting the most out of my available resources without throwing more money at it.

Re: gnu utils and 60's and 70's comments ... Yea, I admit that my knowledge of the gnu utils started in 1990 and I know that unix has been around since the 60's, so my initial post stating that the utils have been around since the 60's and that CPU speed and memory sizes were larger than they actually were back then were incorrect. My bad. ;)

Funny how these 2.0 young bucks consider a C-rewrite extreme, when for many coders is is the lingua franca, the bread and butter. I got started on high-level languages but I've made an effort to get familiar with C because it's small, beautiful, and fast as all hell. Fast is powerful.

Part of knowing a tool is knowing when it's inappropriate. A rewrite in ANY language to "fix" a problem that you don't understand is generally a mistake. Doing things the needlessly complex way doesn't make you a better or manlier programmer.

By the way, a lot of these perl and python scripts spend most of their time calling into C libraries anyway, so the performance difference is often negligible.

And unless my knowledge of the Python standard library is way off, the Python standard library is written in Python, not C. That's what all those .py files are in the /lib folder. So python scripts spend most of their time calling into Python libraries. It's true that the interpreter is written in c (c++?,) maybe that's what you were thinking of.

Why would you bring manliness into the discussion? Are you suggesting that Perl and Python are somehow manlier? Because I have no idea what that means.

Speed comes in two forms, execution time and development and maintenance time, C is only good for one of these. Python or Ruby would be better for the other.

People that complain so much about C never had a change to learn it well. Go talk to Linus Torvalds about changing from C anything else....

I wrote almost exclusively in C for 12 years. I wrote cgiemail, which was very widely used. Today I would never start a server-side web project in C.

Which parts don't I know well?

apples and oranges. Linus writes OS code. Good luck doing your next web 2.0 startup if you write it primarily in C

Why do you think C is such a maintenance issue?

I like it's well accepted that garbage collection is a huge boon for productivity and stability

C has a variety of garbage collectors, here's one:

http://www.hpl.hp.com/personal/Hans_Boehm/gc/

I wonder if it's really more of an intimidation factor than anything (and I'm not saying this to be a jerk, I am truly curious why people have the perception that C is such a nightmare to maintain).

Again, you're looking at best case. Talk to us when GC is mandatory, not optional, because until then, we'll still have to work with C code that can't be trusted because it may not use a GC. To folks who want higher level languages, CG is not an optional feature, it's a core part of the language.

Are you suggesting that you can be as productive in C as I can be in ruby? If not then you answered your own question.

Second, are those garbage collectors widely used in C code and if not why?

Because it is. It's much easier to blow your foot off in C than in Python or Ruby. I think it's pretty well accepted that scripting languages are far more maintainable than C. Sure you can write maintainable C, but that's best case, I think it's more revealing to look at the worse case.

Yeah but C's worse case is still better than most of Perl's, PHP's, etc. best cases. Just as an example.

I don't disagree, btw, that dynamic languages are easier, but I'm really wondering if C's maintenance is really more of a perception than a reality.

That's just an absurd statement. C's worse case is obfuscated garbage http://en.wikipedia.org/wiki/International_Obfuscated_C_Code... that just about no one could read or maintain and you think that's better that Python, PHP, Pearl, or Ruby's best case?

Or is it more likely that like most C folk, you're just stuck on execution speed as the only measure of a language or you don't know what maintainability actually means?

C's being hard to maintain is not a perception, it's a reality. If C were easy to maintain there wouldn't have been room for scripting languages to exist, everyone would have just kept using C. C is extremely low level and extremely powerful, making it hard to work in and very dangerous.

Yes you can do anything with a for loop, but I'd much rather use higher level abstractions like higher order functions, first class functions, objects, continuations, and exceptions and not be able to directly play with memory or pointers because I don't give a crap about the machine, I give a crap about abstracting and solving my problem with the least amount of necessary boilerplate or repetitive code.

Look, make it simple, assume for the sake of argument that Python and C have exactly equal execution speed under all circumstances, given that axiom, why would anyone choose C over Python? Which language do you think most people would choose?

You reveal the primary flaw in your thinking with the phrase "C folk." Languages are not enemy camps. Just learn them both and use the appropriate tool in the appropriate situation - python for glue, prototyping, business systems, web programming, etc. C for operating systems, networking layers, database engines, etc. Try not to attach your ego to either one.

You're hand waving, and there was no flaw in my thinking. C doesn't scare me and I use many languages, and that still has no bearing on the point of the conversation. C is a maintainability nightmare compared to lighter weight scripting languages. That fact stands on its own.

There are "C folk" who won't use anything else because they're stuck in the rut of only thinking of execution speed, pointing that out is not a flaw, it's a fact.

Ok, I will address your argument directly. You state that it's possible to write maintainable C in the best case, but that the worst-case is more revealing. However if we're to be fair, we must also look at the worst case for scripting languages. For example, which of these is more maintainable?

http://www.p-nand-q.com/python/obfuscated_python.html http://en.wikipedia.org/wiki/International_Obfuscated_C_Code...

I would suggest that neither one is particularly maintainable, nor are either of these styles of code likely to appear in the real world. Therefore the worst case for these languages is not really more telling, and not applicable to this discussion.

I wasn't the one who claimed C's worse case was better than Python's best. Perhaps you should reread the thread and see what I was objecting to. You can write bad code in any language, but some don't let you blow your foot off as easily. It's well known and well accepted that higher level languages are more maintainable that C, that's just a fact. If you don't agree, then you're the one making radical claims, not I. Regardless, I'm bored now, this is a dead horse.

I'm curious as to what you are storing besides the URL. Given the two figures, (200M RAM, 30k URLs) I calculate that you are using 7k per URL, which seems a bit excessive to me. Or does the 200M include other data besides the URLs?

Yes, that's the right question to be asking. There's no reason why anything should be taking 7k per url.

fully agree. From one 40 year old hacker to another, C is a very meaningful and useful language. It is not a "premature optimization" for someone who really gets unix/linux. I have built http://shellshadow.com with C and have been very happy with the decision to get "closer to the metal" from the beginning. My server processes are lean and mean. They don't crash for inexplicable reasons. When I have a bug, there are not many layers of technology to slug through.

A server written with boost's asio in C++ will be just as lean and mean, but a fraction of the code.

even if you don't have funding how hard is it to get some old desktops that can have a couple of gigs of memory?

Your urls don't need to be in BDB. Just write them to a text file and stream from that. If you want to check for duplicates, just find a simple bloom filter library. There, now you're under a megabyte. :)

dbserver swf # ps auxw | grep crawl

root 6984 3.0 0.5 215832 5168 pts/5 Sl+ 12:59 0:00 ./crawl

I'm using a linked-list in C to store all of the queued-up URLs so all of the threads can mark them as being worked on or whatever. Also, the app and curl lib takes a few megs too. I'm sure that the heap and multi-threadding libs are responsible for some of the resident memory usage. :) the on-disk binary is only 12k if that make a difference. ;)

The struct:

typedef struct my_url { char * url; int retcode; double old_size; double new_size; status status; struct my_url * next; char * filename; int flashurl_id; int depth; } my_url;

I also like keeping them in memory as opposed to a BDB because of access speed - 25+ threads don't have to wait when it's all in memory, but 25+ threads hitting a BDB would be much nastier on the OS.