I somewhat agree when people say "code reuse has failed". When people set out to write "reusable code", it ends up not being reusable. Usually because they think of reuse as writing a bunch of classes that they can "import".
But writing tools is another way to reuse software. For example, in Unix, you reuse ssh for git and for scp. With the web, you can reuse a huge amount of work in Varnish and nginx by chaining components.
So library reuse is not all it's cracked up to be, but that's not what McIlroy is doing. His solution to the word count problem is fantastically and obviously better. It's real reuse.
To be clear, I think there's a distinction between writing a library for well known use cases. I wouldn't suggest stdio is a waste. But it's almost always a waste of time to emerge from a client project with a bunch of libraries for functionality that will likely never be used again.
And among good developers I think this is one of the biggest problems. I've encountered many times good programmers building elaborate frameworks that take as long as the main project itself, who then present the utility of this framework, which we never use again. They didn't understand the problem space well enough to know when reusability made sense and across what pivots.
So library reuse is not all it's cracked up to be, but that's not what McIlroy is doing. His solution to the word count problem is fantastically and obviously better. It's real reuse.
Knuth also reused. He reused a compiler. He reused input output capabilities. He reused subroutine abstraction capability.
IMO it's less what they used and more what they produced (as all developers reuse). Knuth didn't produce a bunch of reusable tools and I think that is justified and the right approach most of the time.
I went back and reread the paper just now and McIlroy even alludes to this:
"The utilities employed in this trivial solution are Unix staples. They grew up over the years as people noticed useful steps that tended to recur in real problems. Every one was written first for a particular need, but untangled from the specific application.
With time they accreted a few optional parameters to handle variant, but closely related, tasks. Sort, for example, did not at first admit reverse or numeric ordering, but these options were eventually identified as worth adding."
What he describes is the right approach -- and what Knuth wrote is the first step here. I'm fairly certain if Knuth was a systems developer and this came up over and over again you'd see refinement and tools/libraries that made certain aspects of this more reusable. But that's neither his domain nor the specific ask for this client.
McIlroy's rant here seems pretentious in light of this. A much better rant would be to show how Unix tools could in six lines replace LaTex for typesetting (without using anything provided by LaTex).
But it isn't contradicting the point of what McIlroy does -- it actually supports it. Unix tools are more reusable than libraries full of code.
I can't tell what the second part of your post is saying. It doesn't matter if Knuth "would have" done something; McIlroy already did it. The problem is solved with 6 lines. End of story. No pontificating. That's what Unix lets you do -- get on with your day :)
In the sense you're using "library" here, the Unix utilities are libraries--more precisely, they are library functions; the whole set of utilities is the library. So McIlroy was a library/framework writer.
> Most code, even when written with intent to be reusable -- isn't.
I more or less agree with this; but fortunately, you don't have to look at "most" code to find code that is reusable. As witness, once again, the Unix utilities. Or, for that matter, the Python builtins and standard library that I used to build the Python versions of the "word count" program.
Moreover, even if you're in Knuth's position and don't have a good base of reusable code to draw on already, it still makes a big difference how you proceed from there. Do all app devs write big monolithic apps? I believe pg once wrote an essay about bottom-up vs. top-down programming, leaning heavily towards the former.
The essay: http://www.paulgraham.com/progbot.html ?
I think a lot of everyday reuse (by factoring commonly used code) happens at a level of programming which neither fits application nor library development but writing small tools to automate ad-hoc tasks (e.g. by scientists, admins, etc.) Environments like Python, the shell, or Lisps seem to be more suited to this kind of development than, say, Java.
In the case of the example, I could easily imagine that the "isolate words in input text"-part will turn out to show up again, and then one would factor it out to a function. When the first text containing unicode soft hyphen characters is encountered, only a single shared function has to be updated, and all little scripts using it will benefit!
A function extracting the text from xml-documents, skipping a configurable set of element types, might find similar reuse, etc.
Just an aside: I'm not familiar with Python, what I found disappointing was the mix of chained object method invocations and "outer" function calls - since replacing the built-in string class seems to be frowned upon, I was wondering how the example script would look like using pipe[1] :-)
One question about the Python code: is the first call to sorted needed, and if yes, why?
Yes, that's the one I was thinking of. Thanks for finding the link!
> the mix of chained object method invocations and "outer" function calls
Actually, you're right, I could have factored out the part that builds the dict of words vs. occurrences into its own function, so that the entire pipeline would be a single chain of calls. Now that you've put the thought into my head, I may go back and do that. :-)
> I was wondering how the example script would look like using pipe[1] :-)
That would certainly make the Python look more like the shell script. :-)
One of the constraints I imposed on myself when writing the Python version was to only use what comes with it--built in functions/syntax and the standard library, with no third-party packages, similar to how McIlroy only used "built-in" Unix commands that came with every Unix system.
> One question about the Python code: is the first call to sorted needed
You're right, it isn't. I've pushed an update to the github repo fixing this (in both versions).
Just did it. :-) Factored out a "uniq" function in both versions, so the "pipeline" function is now a single chain of method invocations (with the slicing at the end). Which means it could actually be put in the "if __name__ == '__main__'" stanza, but that seems less readable to me.
That I can agree with. But there's nothing stopping you from breaking up your program into small utilities that have a simple interface. Those utilities don't need to be generic or widely applicable. But separating them from the main program can help.
Just to give an example, I once had trouble writing a function that transformed a particular tree data structure. I ended up breaking it in two: the walking part, that took the transformation part as an argument. And of course the transformation part. Note that the walking part was fairly generic, and could have been use for other purposes (heck, one of its arguments where a function!). But I didn't write it for that. I wrote it to reduce the amount of brainpower I would need to finish my program, through dumb separation of concerns.
I liked Hughes paper. Funnily enough, the lack of laziness in Ocaml quite hindered me when I wrote and used my Parsec clone (again in USSM, but in the current version).
Knut's program's most glaring flaw isn't its length, but its monolithic architecture.
In my experience writing reusable code, unless you're writing a library, is usually a waste of time. Must code, even when written with intent to be reusable -- isn't. Maintainability and readability almost always trump reusability. Refactoring fast the point where reuse is needed typically is the better approach.
McIlroys approach is the less common approach of library/framework writers while Knuth is the approach of app devs.