The other thing I absolutely love about pipes is the ease of writing custom utilities. I'd refactored a filetype for use with a game last week, and needed to insert several lines into ~1.5k ASCII files. 1/2 hour to make a hacked c++ program to make the changes, and pass the outputs of grep to that.

Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that.

> Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that.

You didn't miss it, each stage in the pipeline runs in it's own process so you're already using it. With say "grep -r foo ./ | do-something | do-omething-else" the second and third stages will be operating on the data from grep while grep is still running.

In addition to that you can get more parallelism pretty simply with xargs (http://offbytwo.com/2011/06/26/things-you-didnt-know-about-x...). If that's not enough then check out gnu parallel (https://www.gnu.org/software/parallel/).

While we're talking about unix as an IDE I want to put in a plug for inotifywait (https://linux.die.net/man/1/inotifywait). Combining this with a makefile you can get some very fast builds with near instant feedback everytime you save a file. I have one that executes units tests and valgrind every time for instant memory leak detection. Another great use case is getting WYSIWYG like features with latex files.

using make -j $num is another way to go to get parallelism that is sometimes convenient.

Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.

This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages.

To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?

I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?

IIRC make -j is equivalent to make -j256...

From the man page:

> If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.

Although other implementations might be different of course.

It's not a notion that I carried into my implementation of redo, for example.

    % redo --jobs
    redo: ERROR: jobs: missing option argument
    %

What made you go with that rather than trying to pick a sensible default like the number of cores available? Or was it just a general goal to require explicitness?

It was simplicity, of design, implementation, use, and indeed explanation. There's only one default, the default if the --jobs option is not used; rather than there being two defaults, one for not having the option and another for having the option but not the option argument.

More generally, I try to avoid optional arguments to command options, in line with the guideline in the Single Unix Specification. (I actually picked such ideas up many years ago, before the first POSIX specification, from a book by Eric Foxley titled Unix for Super Users, where there was an appendix on command line option syntax.)

* http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...

Of course, if the user wants to use the number of cores available, then xe can work that number out and pass it as an argument; and indeed I have done that very thing in some of the package/make scripts that I have published.

You mean function composition on Lisp Machines REPL and Smalltalk Transcript, with the benefit of structured data.

I've used guile to do pipeline like transformations of data by using multiple function calls. It has the advantage of a more consistent api to do the transformations on the stream of data and surely better type checking. The syntax is different but I think about it as being similar.

Despite its messiness, doing ad-hoc integration of command line tools to transform text is really handy. It also spreads the cpu utilization and memory usage across the commands that you use in the pipeline.

I'll add that I enjoy using Common Lisp and Scheme REPL environments. I also really like Squeak as an interactive programming environment. Neither has replaced the old school Unix shell as my preferred command invocation environment, at least in my professional life.

I have the same experience, and I suspect it's because we haven't replicated the utilities in these environments. No one uses them as a replacement for shell because no one has done the work to use them as a replacement for shell.

Personally, I think Prolog is the right way to go, but I have this idea that I don't want to spend the rest of my life typing into a teletype emulator.

We kind of do on Windows, thanks PowerShell and PowerShell IDE.

> It also spreads the cpu utilization and memory usage across the commands that you use in the pipeline.

For example with Clojure you can use pmap and pcall to achieve that.

That's cool. I'll have to try it out.

Which is your favourite shell? bash, oh-my-zsh, fish?