I do think there's a growing split between server like things and desktop like systems that people actually regularly use. For example, Alpine is super popular in containers, not so much on desktops. Same thing with Busybox, but that's more explicitly meant for embedded envs.
I think that I can present an even better piece of empirical evidence (not to diminish the poignancy of your own):
Almost no large or successful piece of software is composed of Unix command-line utilities.
Firefox, Linux, gcc, Visual Studio Code, Windows, Chrome, Google services, Discord, Spotify, Blender, GIMP, Apache, Slack, Office, Steam, Matrix/Element...almost every single non-trivial piece of software in existence is not composed out of Unix shell utility pipelines.
"Do one thing well" allows you to do one thing well, tautologically - but it doesn't allow you to do many things well. The fact that Unix utilities compose at a very superficial level is irrelevant, because the use of text as an interchange format is such a heart-stoppingly bad idea (consider a programming language where the only type is a string - you wanna write a web server in that?) that it renders Unix tools near-useless for building larger tools (as evidenced by the fact that nothing is made with them), and only useful for immediate interactive use, and building small scripts with (that would already be better served by a real programming language).
The Unix Philosophy is self-contradictory. "Do one thing well" is meant to enable composition, for building larger programs - but "Text is the universal interface" is in direct opposition to building larger programs. But, because tools are only supposed to do one thing well, they're feature-impoverished and not very useful on their own, so you have to compose them. You're screwed either way.
I've written tons of shell scripts for one-off jobs or simple work that would've taken 2, 3, maybe 4 times the amount the time to write in a "real" programming language. (E.g. throwaway code to generate 10,000 test files that are slightly different would take me 5-10 minutes in bash, but maybe half an hour in node or python.)
Passing around strings isn't the BEST paradigm but it's extremely flexible when you need flexibility. There are times when you need something more solid and that's fine too.
And fwiw I'd bet a majority of the software you listed uses bash somewhere in either their build process or SAAS stack.
Moreover, "time to write script" by itself is a bad metric, for several reasons: (1) the cost of understanding and maintaining the script is ignored (and obviously higher for bash than Python) (2) you're not including the higher likelihood of bugs in your bash programs (and all it takes is a single bug that requires 20 minutes to find and fix...) (3) if you had more experience with Python (or with specific libraries useful for writing shell-script-like things - this often goes overlooked) then your development time would probably be shorter and (4) the pool of Python devs is far greater than the pool of bash devs.
> Passing around strings...[is] extremely flexible when you need flexibility
This isn't accurate. Passing around typed data is just as flexible as strings - except that it adds a bunch of extra safety and type-checks. All data in "string" format, with the singular exception of actual English text that is being passed to an NLP system or something, is actually in a structured text format (the operations that you perform on which are merely isomorphic to existing operations on structured data)...that is very likely under-specified relative and more brittle than actual typed data. All that using a string representation does is make errors easier to make and harder to find.
> And fwiw I'd bet a majority of the software you listed uses bash somewhere in either their build process or SAAS stack.
Yes, because bash is a (bad) habit for tooling devs, not because it scales well or adds safety/performance/maintainability - as evidenced by the fact that there aren't any large pieces of software written in it.
That is - bash is used "somewhere" and not "everywhere" because you can only get away with it at small scales - at large scales, it just falls apart.
All true.
However, the development environment (and frequently the build systems) used for all of the above are likely substantially composed out of Unix shell utilities and pipelines, aided by very unix-y tools such as Perl or awk.
That says something, I think. Just not sure what.
One of the possibilities is "just legacy code, bad habits, and bandwagoning", but that's a bit of a cop-out.
Another is "Unix utilities have a much better API to the filesystem and invoking other programs than most languages available today" - which is a (somewhat more-) concrete issue that we can address in other languages!
A third possibility is that the Unix tools transition very well from interactive use to batch use in a larger program - again, something that we can work on improving in other languages. Emacs seems to also do this - functions that are usable in both interactive mode (either invoking with M-x or through a keybinding) and as pieces in larger scripts/functions are pervasive. Breck Yunits calls these "user methods"[1] and I think that it's a good idea for a great pattern.
No, it is not. You said "wrong-headed" and then made a very vague analogy. Your point is the opposite of "clear" - in fact, it's unclear whether it exists at all.
> Your criticism of UNIX commands is not valid in the same way that it not a valid criticism of garage tools that major vehicle manufacturers do not build cars out of garage tools.
That's not a valid argument, either. That's a flawed, leaky, vague analogy that does nothing to actually describe, concretely, what your point is. To summarize: you made a non-argument, called it "clear", and then made the same non-argument again.
Here are my points, concretely: the fact that no major application has ever been built even mostly out of Unix shell very strongly indicates that it's unsuitable as a programming language, in addition to indicating its unpopularity. The fact that the most popular variants of Unix shell do not adhere well to the Unix philosophy also indicate that the Unix philosophy isn't a very good one for interactive use, either. The fact that all of the Unix shells (whether Unix-y or not) aren't used at all for building large applications also indicates that the Unix philosophy isn't suitable for building large programs.
> My understanding is major applications aren't made using UNIX commands because they are ordinarily developed using programing language.
The Unix shells are programming languages that implement the Unix philosophy. The fact that they aren't used to build any large or particularly useful piece of software means that the Unix philosophy itself isn't useful for those things.
> a straw man fallacy in the precise way it is fallacious if I observe as a criticism of UNIX tools that no major coffee manufacturers make their product with UNIX tools.
You used a lot of fancy words here to conceal the fact that your point is completely invalid. There's no relationship between coffee and Unix tools, but there's a very strong relationship between the Unix philosophy and the Unix tools. Therefore, the Unix tools not being used to make large and useful systems directly implies that the Unix philosophy is bad at the same.
Moreover, one of the selling points of the Unix tools is that you can use them as a programming language and compose them to make larger programs - so, the fact that people do not do that is, indeed, a valid and cogent criticism of them.
Posix shell is a programming language where the only type is a string. Text as an interchange format works reasonably well in practice for a wide range of tasks.
To me shell, easy to combine unix commands and text as interchange is a clear example of New Jersey style from "Worse is better" [1] and their critique comes mostly from followers of MIT/Stanford style.
In my carrier I wrote hundreds of small shell scripts which use unix commands and text as an an interchange format. Yes, I see that some bugs can be prevented/detected by using something more strongly typed than shell, but all my scripts worked reliably enough and didn't suffer from any significant bugs so why bother? (disclaimer: I'm old fashioned and like to read man pages; I imagine shell can be full of traps if you ignore documentation and use only trail and error approach or practice stackoverflow driven development).
Of course for large projects and even some small ones other tools/languages fit better than shell but shell have its niche.
You got it!
> Text as an interchange format works reasonably well in practice for a wide range of tasks.
Solutions using real programming languages score better on almost every conceivable metric (robustness, tooling, debuggability, maintainability, extensibility, performance, library ecosystem, developer pool, language pitfalls) other than initial time-to-develop, where they only win by a small margin (e.g. 10 minutes to develop a script instead of 2) - so, unless you're developing software purely for your own enjoyment, it's somewhat irresponsible to use shell instead of Python (or some other real, but script-amenable, programming language).
> To me shell, easy to combine unix commands and text as interchange is a clear example of New Jersey style from "Worse is better" [1] and their critique comes mostly from followers of MIT/Stanford style.
The New Jersey style is not only flawed on a theoretical level (just go look at any of those characteristics - tell me that it's more important for your bank or X-ray machine software, or your compiler or firewall to be simple than correct - and it's pretty clear that software that prioritizes simplicity above all else doesn't actually usefully serve humans, which is the point of software in the first place) but also unpopular on a practical level for actually building useful pieces of software.
The New Jersey style is a style guide that is only useful for making art projects (where the value is in beauty, not utility) that aren't required to be performant, maintainable, feature-complete, or have correct behavior.
> all my scripts worked reliably enough and didn't suffer from any significant bugs so why bother
Because of the above list of metrics that real languages score better than shell on - the only thing that shell wins on is initial time-to-development! It's going to be harder for others to maintain, debug, optimize, understand, and extend your scripts after you leave, harder to diagnose bugs, harder to find people who can write bash...
> shell have its niche
Yes - but it's an extremely narrow one where time-to-development is the most important metric and you're only working with small problems (as above a relatively low problem complexity, shell scripting is slower than using real languages anyway) - and these conditions are very rarely true at the same time (e.g. in infrastructure you might have a small problem, but correctness is far more important than TTS - while in a startup you want low TTS, but your problems are non-trivial).
Also Powershell doesn't get much love from UNIX heads.
Do one thing well and text as the universal interface works well in the small for parsing files, doing sys admin tasks, gluing a few things together. That's what it's intended for. It's obviously the wrong tool to build a system in. Used in the right context they compose beautifully.
Or perhaps those BSDers, suckless tool users, etc get setup and quietly just get to work. I feel that’s plausible.
I think you’re right re: server/desktop split, which decades ago might have been hard to tell apart. Is desktop-computing and some loud but ignorant (of Unix) majority directing the future of Unix down the garden path or are the traditionalists out of touch and just yelling at clouds and kids to stay off their lawn?
I fall into the traditionalist camp - I think the Unix that got us here is Good and to know it is to love it.
I think pushing back against complexity is (and has been for a long time) an additional challenge for BSD/Linux, besides it’s primary role of being a great compute environment. The BSDs have done marvellously here.
> I mean the empirical evidence alone seems point at the Unix Philosophy not actually being popular in practice.
I’m worried this is polling 100 people off the street about how they care for their kitchen knives and trying to tie that back to Michelin Star restaurant kitchens. I don’t know…
The only big UNIX where I actually did install GNU stuff was on Solaris 2.6 via the GNU packages repository, because I was requested to do so.
All our HP-UX and Aix boxes used the vendor tooling (including the C compilers).
Fortunately that didn't work when you needed 3D graphics hardware. But I guess you worked in a 'server' company, not a graphics (3D or engineering) one?
> All our HP-UX and Aix boxes used the vendor tooling (including the C compilers).
Of course, GCC was only needed for most of the later (early 2000s) OSS like Mozilla that didn't work with the native compilers because of missing GNU extensions. That's one thing I don't miss - the prices of compilers. SGI took more than 10,000 € for their MipsPro C, C++ and Fortran compilers but without 'advanced' optimization options like auto parallelization and their 'IDE' tools, Developer Workshop. That reminded my of Totalview (a debugger from ???): now Perforce bought them https://totalview.io/
Or SGI Onyx, for another example.
In fact the whole cloud streaming hype is basically a bunch of people rediscovering graphics programming with timesharing on UNIX.
Actually one way to mix both, while controlling what people were allowed to do on their workstations was to boot via tftp and have $HOME mapped noexec over NFS.
A solution used in some labs at the university campus.
That quote from the Uncomfortable Truths article is strange because Dan Luu doesn't explain that "it doesn't actually work well". He just explains that the philosophy is not consistently followed, but he is not unhappy with it.
Which is exactly the reason for the glut of command line options.
A great example of following Unix philosophy using a more recent example is the ssh client on plan 9. It is split into three separate programs each with their own man page: ssh(1) for terminal access, sshfs(4) for mounting a remote file tree, and sshnet(4) which imports a remote machines tcp stack. Keep it simple, stupid.
I disagree. Yes, nobody uses sendmail anymore, filesystems vastly improved and X isn't a resource hog any more. I've reread it some months ago and the main points still hold.
For example the part about `kill`:
Most operating systems have a “kill” command. So does Unix. In
most operating systems, the kill command kills a process. The Unix
implementation is much more general: the “kill” command sends a
process a message. This illustrates the first Unix design principle:
• Give the user power by making operations fully general.
The kill command is very powerful; it lets you send all sorts of messages to
processes. For example, one message you can send to a process tells it to
kill itself. This message is -9. -9 is, of course, the
largest single-digit message, which illustrates another important
Unix design principle:
• Choose simple names that reflect function.I think the UUHB is mostly worth reading to understand the absolutely dismal state of proprietary Unix in the 90s. It explains why technical reasons contributed to the growth of both Windows NT and GNU.
"Those days [of the Unix Philosophy] are dead and gone, and the eulogy was delivered by Perl."
https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
there's a subtle thing about perl that's not mentioned often, which is how it borrowed ideas from many other tools and that made perl "intuitive" to learn for people who knew the other tools. But once you're embedded in just perl for awhile, you forget the other tools and then perl's grab-bag of borrowed ideas becomes sort of burdensome, it loses "intuitiveness".
If anything, Perl killed classic Unix and much of the Unix philosophy.
Every major modern design looks more like Perl than like Unix: Python, Ruby, PHP, JavaScript, Java, C#, Swift, Go, Rust, etc.
They're all "batteries included".
Perl's syntax is a relic of its time, and it blazed a trail for what to do (and what to avoid!) in later scripting languages.
Back in the mid-1990's Apple once promoted OpenDoc (https://en.wikipedia.org/wiki/OpenDoc), a standard that was envisioned to support an ecosystem of component-based software, where users could mix and match components to create modular solutions instead of depending on large, monolithic software applications. Here is a nice short video promo from roughly 1994 describing OpenDoc: https://www.youtube.com/watch?v=oFJdjk2rq4E.
OpenDoc was nixed when Steve Jobs returned to Apple; my opinion for this nixing is because (1) Apple at the time needed to focus on one technical direction (OpenDoc was one of many competing software visions at Apple before Steve Jobs united Apple toward a software vision built on top of OpenStep/Cocoa), and (2) it's hard to promote a component-based software ecosystem when the Mac was on life support and needed support from popular vendors of large, monolithic applications (I'm talking mainly about Adobe and Microsoft) in order for Mac users of these applications to stay on the Mac. I believe this is the same reason why we haven't seen much of a component-based software ecosystem on Windows outside of Microsoft Office's interoperability among its applications: it's easier for software vendors to sell integrated solutions than to sell components that users will have to integrate themselves.
I wish the FOSS desktop ecosystem, which doesn't have the same commercial concerns as the world of proprietary software, embraced component-based software beyond Unix pipes. However, the FOSS desktop ecosystem rallied behind KDE and GNOME in the latter half of the 1990's, and thus many of us still rely on large applications like LibreOffice, Firefox, and the GIMP instead of component-based alternatives.
[0] https://en.wikipedia.org/wiki/Desktop_communication_protocol
There is also Vala from Gnome:
The biggest pain with COM is that Microsoft (except for C++/CX) never put much effort like Borland did, or VB 6, on making it a pleasant experience.
.NET was supposed to replace it, but WinDev rebeled against it, so now we have both, with COM still having the tooling stuck in 2000 (C++/WinRT is a regression to those days).
On UNIX side, there are a couple of possibilities gRPC, D-BUS, DCOP, KParts, Android Binder/Intents,...
macOS builds on the NeXT Tooling and also offers XPC in addition to them.
However where BSD and Linux clones fail is having a full stack approach to the platform, thus turning such approaches in mini-silos, e.g. you can enjoy D-BUS alongside KParts as long as you stay with KDE and KDE specific apps.
> For example, Unix programs often have static tables or fixed-size strings, which make for arbitrary limits; use dynamic allocation instead.
> Avoid arbitrary limits on the length or number of any data structure, including file names, lines, files, and symbols, by allocating all data structures dynamically. In most Unix utilities, “long lines are silently truncated”. This is not acceptable in a GNU utility.
> The Unix philosophy of “do one thing well” doesn’t actually work that well. Dan Luu explains this better than I could.
Usually The Unix Philosophy gets nothing but praise in these parts so I was interested to see a counterpoint.