A simplified example: If I crawl a website, the chances are good that I'll pull several thousand copies of the same url from the site over and over again. I want to insert all URLs that I find back into the database so I can can crawl them, but I don't want any dupes. I just want to crawl the URL once and then move on. If I insert a URL into a DB and let the DB check if it's a dupe, then I waste DB time/IO. My solution was to crawl thousands of pages, dump all of the urls into a text file, run the text file through 'sort | uniq' and then dump those URLs into the DB and let the DB ignore the dupes at that level. I still have to use a DB to do some of the work, but pre-processing the data up-front using 'sort | uniq' is mega-speedier in my special case.
Also, I did wrap most of the scripts with perl to talk to mysql and do other sanity-checking that's easier in perl than bash or tcsh. :)
Shell scripts are hard to make robust. I don't understand why you wouldn't just drive the unix utilities from perl.
Why use perl threads on unix? Everyone know they suck. Why not fork and use a transactional database for IPC?
I remain unconvinced of the alleged wins people have with flat file solutions. It's usually about replacing a toy like mysql or some convoluted berkelydb mess. You can use db2 for free up to 2 gigs of memory. Why waste even a minute replicating transactional RDBMS functionality by hand? As soon as you're dealing with flock and company and you COULD be using a DB, you should, as far as I'm concerned.