And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them.
There are two uses of the Unix “api”:
[A] Long lived tools for other people to use.
[B] Short lived tools one throws together oneself.
The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools designed for others to use, in A.
The ubiquity of the C APIs of course solved [A] use cases in the past, when it was unconscionable to operate a system without cc(1). It’s part of why they get first class treatment in the Unix man pages, as old fashioned as that seems nowadays.
And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.
Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.
But when you realize that space isn't a valid separator between token, seeing things like "Class to Pull Info From Database::Extractor Tool" actually becomes much easier to read and the language becomes highly expressive, helped somewhat by the insane integration into the firm's systems.
I was on your side until I tried it, but it can actually be quite useful, esp. if everything is consistent.
On my side? You cannot imagine how extreme one can be... If I had my way, programming languages would only allow single-letter variables and space would be the multiplication operator.
Math notation being dependent on single-character names is an artifact of "concatenation as multiplication" -- which in my opinion is a useful but sloppy historical convention with little merit or relevance in non-handwritten media.
Unix tools standardized a character representation (e.g. everything is ascii). The next step is a unified standard the syntactical representation.
I don’t why POSIX tools don’t already do this but I suspect it’s because most tools would have evolved by accident rather than being designed from the ground up with edge cases in mind.
That's potentially even worse in a file name than plain old space is. It looks like a normal space, but isn't.
If you are going to replace spaces in file names with something else, underscores are the best option I think.
Why allow them? And they potentially pose security issues.
With some, admittedly somewhat obscure [1][2], terminal emulators, C0 and C1 controls can even be used to execute arbitrary code. You could say these terminal emulators are insecure, and you may well be right, but the fact is they exist.
[1] See for example page 211 of https://static.zumasys.com/zumasys/atfiles/manuals/at7/AccuT... which documents the ESC STX < command CR escape sequence which tells the AccuTerm terminal emulator to run an arbitrary command
[2] Also the APC C1 control can make Kermit run arbitrary commands, if you've executed "SET APC UNCHECKED" (which is not the default setting) – see http://www.kermitproject.org/onlinebooks/usingckermit3e.pdf page numbered 300 (page 310 of PDF)
Using a mount option and/or filesystem volume flag means it works fine with legacy volumes which might contain those bad names. On such legacy volumes, either that volume flag is not supported by the filesystem, or else it is but is disabled for that particular volume. If you mount it with the mount option, you can still access bad names on that volume, you just can't create new ones; if you need to create new bad names, you just (re)mount it with the mount option disabled.
Also, some languages do allow whitespace in variable names like R and SQL, so long as the variable names are quoted or escaped properly.
It's like people saying you don't need to escape SQL values because they come from constants. Yes, they do... today.
It's not just quoting either. It's setting the separator value and reverting it correctly. It's making sure you're still correct when you're in a function in an undefined state. It's a lot of overhead in larger projects.
For example, there is no way to store the `a b "c d"` in a regular variable in a way where you can then call something similar to `ls $VAR` and get the equivalent of `ls a b "c d"`. You can either get the behavior of `ls a b c d` or `ls "a b c d"`, but if you need `ls a b "c d"` you must go for an array variable with new syntax. This isn't necessarily a big hurdle, but it indicates that the concepts are hard to grasp and possibly inconsistent.
var=( a b "c d" )
ls "${var[@]}"
POSIX shell can get you what you want with eval: var='a b "c d"'
eval "ls $var"
There is also one array variable in POSIX shells, the argument list: set -- a b "c d"
ls "$@" $ exec(‘ls’, ‘-l’, ‘A B C’)
Maybe that’s unrealistic? I mean, if the shell was like that, it probably wouldn’t have exec semantics and would be more like this with direct function calls: $ ls(ls::LONG, ‘A B C’)
Maybe we would drop the parentheses though — they can be reasonably implied given the first token is an unspaced identifier: $ ls ls::LONG, ‘A B C’
And really, given that unquoted identifiers don’t have spaces, we don’t really need the commas either. Could also use ‘-‘ instead of ‘ls::’ to indicate that an identifier is to be interpreted locally in the specific context of the function we are calling, rather than as a generic argument. $ ls -LONG ‘A B C’
If arguments didn’t have spaces, you could make the quotes optional too.QED
The much bigger problem is that spaces make text-only commands compose badly.
$ ls -l `find ./ -name *abc*`
Is going to work very nicely if file names have certain properties (no spaces, no special chars), and it's going to break badly if they don't. Quoting is also very simple in simple cases, but explodes in complexity when you add variables and other commands into the mix. ls -l ${find ./ name *abc*}
# return the result as a single string
ls -l @{find ./ name *abc*}
# return the result as an array (like Bourn shell)
So in the first example, the command run would look something like: ls -l "foo abc bar"
whereas in the second it would behave more like Bourne shell: ls -l foo abc bar
As an aside, the example you provided also wouldn't work in Bourne shell because the asterisks would be expanded by the shell rather than `find`. So you'd need to quote that string: ls -l `find ./ -name "*abc*"`
This also isn't a problem in murex because you can specify whether to audo-expand globbing or not.With POSIX shells you end up with quotation mark soup:
ls -l "`find ./ -name '*abc*'`"
(and we're lucky in this example that we don't need to nest any of the same quotation marks. It often gets uglier than this!)Murex also solves this problem by supporting open and close quote marks via S-Expression-style parentheses:
ls -l (${find ./ -name (*abc*)})
Which is massively more convenient when you'd normally end up having to escape double quotes in POSIX shells.---
Now I'm not trying to advocate murex as a Bash-killer, my point is that it's specifically the design of POSIX shells that cause these problems and not the way Unix pipelines work.
Pipelines to the rescue. If you used a pipeline instead of shell expansion, it can deal with spaces just peachy.
$ find . -name "*abc*" -print0 | xargs -0 ls -l
There's no argument that bash (and other shells) make dealing with spaces and quotes troublesome. That has nothing to do with pipelines.My point wasn't specifically about find, but about the problems of text and other unstructured input/output in general.
Update: reading a bit around the internet, it looks like it's not possible to do something like this :
find / -print0 | \
grep something | \
xargs -0 cat
As grep will mangle the output from find -print0.It's like there is a shortcut that some people use that you want to wall in because it doesn't look pretty to you.
In most other cases I’ve never really had a problem with “this is a place where spaces are ok” (e.g. notes, documents, photos) and “this is a place where they are not ok” — usually in parts of my filesystem where I’m developing code.
It’s fine to make simplifying assumptions if it’s your own code. Command history aside, most one liners we type at the shell are literally throwaways.
I think I was clear that they aren’t the only category of program one writes and that, traditionally on Unix systems, the counterpart to sh was C.
The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.
If you control the inputs and you need to support quotes, spaces, non-white space delimiters, etc... in shell script, then that’s on you.
If you don’t control the inputs, then shell scripts are generally a poor match. For example, if you need summary reports from a client, but they sometimes provide the table in xlxs or csv format — shell might not be a good idea.
Might be controversial, but I think you can tell who works with shell pipes the most by looking at who uses CSV vs tab-delimited text files. Tabs can still be a pain if you have spaces in data. But if you mix shell scripts with CSV, you’re just asking for trouble.
You can define a field separator in the command line with the environment variable IFS - i.e. 'IFS=$(echo -en "\n\b");' for newlines - which takes care of the basic cases like spaces in filenames/directory names when doing a for loop, and if I have other highly structured data that is heavily quoted or has some other sort of structure to it, then I either normalize it in some fashion or, as you suggest, write a perl script.
I haven't found it too much of a burden, even when dealing with exceptionally large files.
Also Zsh solves 99% of my "pipelines are annoying" and "shell scripts are annoying" problems.
Even on systems where I can't set my default shell to Zsh, I either use it anyway inside Tmux, or I just use it for scripting. I suppose I use Zsh the way most people use Perl.
There is a world of stuff in between "I need relatively low-level memory management" and "I need a script to just glue some shit together".
For that we have Python and Perl and Ruby and Go, or even Rust.
(My point about C was in the historical context of Unix, which is relevant when talking about its design principles.)
My source for the claim itself is from 14m30s of Brian Cantrill's talk "Is It Time to Rewrite the Opersting System in Rust" https://youtu.be/HgtRAbE1nBM
Stable tools designed for oneself
Source: I wrote a shell and it solves a great many of the space / quoting problems with POSIX shells.
[0] Really just input + exec + pipes.
[1] https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text
This is what my shell (https://github.com/lmorg/murex) does. It defaults to using JSON as a serialisation format (that's how arrays, maps, etc are stored, how spaces are escaped, etc) but it can re-serialise that data when piping into executables which aren't JSON aware.
Other serialisation formats are also supported such as YAML, CSV and S-Expressions too. I had also considered adding support for ASCII records but it's not a use case I personally run into (unlike JSON, YAML, CSV, etc) so haven't written a marshaller to support it yet.
I have no idea why I am the only person I know who uses them regularly.
- They can be as easily edited by hand like other serialisation formats which use printable characters as their deliminators
- It's not clearly defined how you'd use them for non-tabulated data (such as JSON, XML, S-Expressions)
- There isn't any standard for escaping control characters
- They're harder to differentiate between unserialised binary formats
- They can't be used as a primitive like JSON is to Javascript and S-Expressions is to Lisp.
And if we're both honest, reducing the serialisation overhead doesn't gain you anything when working in the command line. It's a hard enough sell getting websites to support BSON and at least there, there is a tangible benefit of scale.
Not that I'm dismissing ASCII records, they would have been better than the whitespace mess we currently have in POSIX shells. However I don't agree ASCII records are better that current JSON nor S-Expressions.
* http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-...
luck, or simply execute the steps you want to check by typing them on the command line. I find the pipes approach incredibly powerful and simple because you can compose and check each step, and assemble. That's really the point, and the power, of a simple and extensible approach.
echo 'Hi!' | tee "$(printf "foo\nbar\tqu ux.txt")"
cat foo$'\n'bar$'\t'qu' 'ux.txt
IIRC the latter only works in ksh93+, but the former should be fine pretty much in any shell....if it was the 1970s. Nowadays we use Python etc for tools where it’s less acceptable for them to fall apart, though everything falls apart at some point, spaces or otherwise.
The maintainer responded by saying, basically, that the behavior was deeply baked in and was not going to change.
Some things across different systems have pitfalls as a result of their implementation, or some features have pitfalls. There are half a dozen things I can point to on windows that are worse than the "spaces in filenames". Hell, windows doesn't even support ISO 8601 dates, or times to be properly placed in files. It restricts symbols much more than Linux / Unix
echo "foo bar fnord" | cut --delimiter " " --output-delimiter ":" -f1-
# foo:bar:fnord --output-delimiter ":"
Very cool!However it does require the GNU version of cut, not the Mac OS X supplied BSD version. In zsh at least, you can do:
alias cut='/usr/local/bin/gcut'I agree it's messy but not that hard. Just set IFS to null.
XML? SMH.
Well, that's easy. Don't escape or quote the strings; encode them. Turn them into opaque tokens, and then do Unix things to the opaque tokens, before finally decoding them back to being strings.
There's a reason od(1) is in coreutils. It's a Unix fundamental when working with arbitrary data. Hex and base64 are your friends (and Unix tools are happy to deal with both.)
Can you give some examples?
Everything is a byte stream. Usually that means text but sometimes it doesn't. Which means you can do fun stuff like:
- copy file systems over a network: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm...
- stream a file into gzip
- backup or restore an SD card using `cat`
See one here: https://unix.stackexchange.com/questions/6852/best-way-to-re...
Researching, it appears that there are multiple attributions, because it's just that obvious and necessary a routine.
https://www2.cs.duke.edu/csl/docs/unix_course/intro-98.html
https://unix.stackexchange.com/questions/6804/what-does-dd-s...
Could have used a \s or something to clarify.
But yes, that’s the typical file / block management tool. Just remember to set a block size otherwise you’d suffer from worse performance than using cat
Those damn kids with their loud rockn'roll music and their Windows machines. Back in my day we had he vocal stylings of Dean Martin and the verbal stylings of Linus Torvalds let me tell ya.
Seriously though, I'm actually seeing younger engineers really taking the time to learn how to do shell magic, using vim, etc. It's like the generation of programmers who started up until the late 90s used those by default, people who like me started 15-20 years ago grew up on IDEs and GUIs, but I've seen a lot of 20-30 something devs who are really into the Unix way of doing things. The really cool kids are in fact doing it.
Add to all that the fact that the command line allows for a lot of really easy composition and automation, and it's significantly better for me. I can hardly function on a Windows computer!
For projects where I must work in Windows my secret weapon is WSL + tmux + fish + vim for everything without a strict Visual Studio dependency.
Copy-paste within tmux works just fine with (a) set -g mouse on in ~/.tmux.conf (b) mouse select to copy and (c) Ctrl + B ] to paste.
Copy-paste from tmux to Windows or vice versa is a pain.
I haven’t tried all the terminals yet but what I need is an iTerm2 [1] for Windows.
[1]: https://iterm2.com
Giving up the idea that CLI = pain (i.e figuring out how to to navigate a file system, ssh keys, etc) for sure was a learning curve, but now I can't imagine using computers without it.
Not at all, you can pipe around all the binary you want. Until GNU tar added the 'z' option, the way to extract all files in a tarball was:
`gunzip -c < foo.tar.gz | tar x`
However, "text files" in Unix do have a very specific definition, if you want them to work with standard Unix text manipulation utilities like awk, sed, and diff:
All lines of text end are terminated by a line feed, even the last one.
I can't tell you how many "text files" I run across these days that don't have a trailing newline. It's as bad as mixing tabs and spaces, maybe worse.
While posix and streams are nice (if you squint, files look like a poor man’s version of objects imposed on top of bytestreams), it’s about time we moved on to better things, and not lose sight of the bigger picture.
Objects aren't human readable and if they get corrupted the object's format is very difficult to recover, you need an intimate knowledge of both the specific format used (of which there might be thousands of variations). Plaintext, however, if that gets corrupted it's human-readable. The data might not be more compact (Although TSV files tend to come out equal), but it's more robust against those kinds of changes.
Have you ever examined a protobuf file without knowing what the protobuf structure is? I have, as part of various reverse-engineering I did. It's a nightmare, and without documentation (That's frequently out of date or nonexistent) it's almost impossible to figure out what the data actually is, and you never know if you've got it right. Even having part of the recovered data, I can't figure out what the rest of it is.
> Have you ever examined a protobuf file without knowing what the protobuf structure is?
Are you referring to an ASCII serialization or a binary serialization?
In a typical scenario, your application needed structured data (hence protobuf), and then you serialized your protobuf into an ASCII bytestream... so how are you ever worse off with a protobuf compared to a text file, provided you make a good choice of serialization scheme? Clearly text files cannot carry enough structure, so we have to build more layers of scaffolding on top?
Ultimately files are serialized on to disk as bits, and “ASCII” and “posix” are just “plugins” to be used when reading bitstreams (serialized file objects). Plaintext is not readable if the bits get corrupted or byte boundaries/endianness gets shifted!
To have the same amount of of robustness with higher-order structures I imagine all one needs is a well-defined de/serialization protocol which would allow you to load new objects into memory? Am I missing something?
It is allowed, you can pass around data in whatever encoding you desire. Not many do though because text is so useful for humans.
> You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay)
That's a daemon/service and basic unix utils make it easy to create, a shell script reading from a FIFO file fits this definition of object, the message passing is writing to the file and the object is passed around with the filename. Unix is basically a fulfillment of Alan Kay's idea of OO.
Well, it is allowed. You can of course serialize some data into a structured format and pass it over a byte stream in POSIX to this end, and I think this is appropriate in some cases in terms of user convenience. Then you can use tools like xpath or jq to select subfields or mangle.
> (if you squint, files look like a poor man’s version of objects imposed on top of bytestreams)
If all you have is a hammer...
You were inaccurate on the other hand - what is similar to PowerShell ?
It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist.
UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits.
To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that does anything high-level such as processing "structured" data must have a parsing step on its input, and then a serialization step after its internal processing.
The myth of UNIX pipes is that it works like this:
process1 | process2 | process3 | ...
The reality is that physically, it actually works like this: (process1,serialize) | (parse,process2,serialize) | ...
Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This is forced by the use of byte streams to connect each step. It cannot be circumvented! It's an inherent limitation of UNIX style pipes.This is not a limitation of PowerShell, which is object oriented, and passes strongly typed, structured objects between processing steps. It makes it much more elegant, flexible, and in effect "more UNIX than UNIX".
If you want to see this in action, check out my little "challenge" that I posed in a similar thread on YC recently:
https://news.ycombinator.com/item?id=23257776
The solution provided by "JoshuaDavid" really impressed me, because I was under the impression that that simple task is actually borderline impossible with UNIX pipes and GNU tools:
https://news.ycombinator.com/item?id=23267901
Compare that to the much simpler equivalent in PowerShell:
https://news.ycombinator.com/item?id=23270291
Especially take note half of that script is sample data!
> Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material.
UNIX is entirely too cool with young people. They have conflated the legacy UNIX design of the 1970s with their preferred open source software, languages, and platforms.
There are better things out there, and UNIX isn't the be all and end all of system management.
This can also be a security concern. According to research, a great number of defects with security implications occur at the input handling layers:
The maintainer apparently has no transport security concerns. It takes literally two minutes to set up TLS these days
The reality is that physically, it actually works like this:
(process1,serialize) | (parse,process2,serialize) | ...
But as soon as you involve the network or persistent storage, you need to do all that anyway. And one of the beauties is that the tools are agnostic to where the data comes from, or goes.Look up what the Export-Csv, ConvertTo-Json, and Export-CliXml, Invoke-Command -ComputerName 'someserver', or Import-Sql commands do in PowerShell.
Actually, never mind, the clear and consistent naming convention already told you exactly what they do: they serialise the structured pipeline objects into a variety of formats of your choosing. They can do this with files, in-memory, to and from the network, or even involving databases.
This is the right way to do things. This is the UNIX philosophy. It's just that traditional shells typically used on UNIX do a bad job at implementing the philosophy.
Most data being line oriented (with white space field separators) has historically worked well enough, but I get your point that the wheels will eventually fall off.
It’s important to remember that the shell hasn’t always been about high concept programming tasks.
$ grep TODO xmas-gifts
grandma TODO
auntie Sian TODO
jeanie MASTODON T shirt
The bug (bugs!) in the above example doesn’t really matter in the context of the task at hand.(7) and common signalling facility
(8) also nice if some files have magic properties like /dev/random or /proc or /dev/null
(9) every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors
This is false - if a process closes stdin/out/err, its children won't have these files open.
That said, it's still a bit of a pedantic point, you can expect to always have those three fds in the same way you can expect `argv[0]` to represent the name of your executable (Which also isn't enforced by the kernel).
The privileged program opens a sensitive data file for writing, it gets assigned (say) file descriptor 2, because someone in a parent process took some advice to "close your standard I/O because you are a dæmon" to heart, other completely unrelated library code somewhere else blithely logs network-supplied input to standard error without sanitizing it, and (lo!) there's a remotely accessible file overwrite exploit of the sensitive data file.
Ironically, the correct thing to do as a dæmon is to not close standard I/O, but to leave setting it up to the service management subsystem, and use it as-is. That way, when the service management subsystem connects standard output and error to a pipe and arranges to log everything coming out the other end, daemontools-style, the idea actually works. (-:
The point of separate error streams echoes the main Unix philosophy of doing small things, on the happy path, that work quietly or break early. Your happy results can be chained on to the next thing and your errors go somewhere for autopsy if necessary. Eg,
p1 | p2 | p3 > result.txt 2> errors.txt
will not contaminate your result file with any stderr gripes. You may need to arrange error streams from p1 and p2 using () or their own redirection. If it dies, it will die early.BUT speaking of logic, the concept is close to Railway Oriented Programming, where you can avoid all the failure conditionals and just pass a monad out of a function, which lets you chain happy path neatly. Eg, nice talk here https://fsharpforfunandprofit.com/posts/recipe-part2/
2. Everything is a file descriptor
3. File descriptors are things that support standard I/O system calls such as read and write
I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do:
cat < /dev/ttyS0
to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80
cat < /tmp/mysocket
It is weird that so many things in Unix are somehow twisted into files (/dev/fb0??), but network connections, to my knowledge, never managed to go that way. I know we have netcat but it's not the same.That's a whole rabbit hole to go down, but Plan 9 networking is much more like your hypothetical example [0]. Additionally, things like the framebuffer and devices are also exposed as pure files, whereas on Linux and most unix-like systems such devices are just endpoints for making ioctl() calls.
For networking, I don't think there's a particular reason it doesn't exist, but it's worth noting sockets are a little special and require a bit extra care (Which is why they have special syscalls like `shutdown()` and `send()` and `recv()`). If you did pass a TCP socket to `cat` or other programs (Which you can do! just a bit of fork-exec magic), you'd discover it doesn't always work quite right. And while I don't know the history on why such a feature doesn't exist, with the fact that tools like socat or curl do the job pretty well I don't think it's seen as that necessary.
/dev/ttyS0 should really be /dev/ttyS0/{data, ctr, baud, parity, stopbits, startbits} (probably a bunch I'm forgetting).
exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.google.com\r\nConnection: close\r\n\r\n" >&3
cat <&3
https://news.ycombinator.com/item?id=23422423 has a good point that "everything is a file" is maybe less useful than "everything is a file descriptor". the shell is a tool for setting up pipelines of processes and linking their file descriptors together. printf 'GET /%s HTTP/1.0\nHost: %s:%s\n\n' "${2-}" "${1-0}" "${3-80}" |
/usr/local/bin/tcpclient -RHl0 -- "${1-0}" "${3-80}" sh -c '
/usr/local/bin/addcr >&7
exec /usr/local/bin/delcr <&6
' |
awk '/^$/ { body=1; next } { if (body) print }'Additionally, there is /dev/tcp in bash.
>(2) everything (ish) is a file
I'm having a moment. I have to get logs out of AWS, the thing is frustrating to no end. There's a stream, it goes somewhere, it can be written to S3, topics and queues, but it's not like I can jack it to a huge file or many little files and run text processing filters on it. Am I stupid, behind the times, tied to a dead model, just don't get it? There's no "landing" anywhere that I can examine things directly. I miss the accessibility of basic units which I can examine and do things to with simple tools.
Yes, it’s very annoying I have to do all that, but I commend Bezos for his foresight in demanding everything be driven by APIs.
(1) you duplicate yourself with fork()
(2) the parent process continues about its business
(3) the created process calls execve() to become something else
This is the mechanism by which all new processes are created, from init (process 1) to anything you run from your shell.
Real life examples here, in this example of a Unix shell:
plenty of basic examples with execv, execve, execvp, etc.
if the execv() call succeed then it completely replaces the program running in the process that calls it
you can do plenty of tricks with that ;)
Yes, I know. You've probably never touched either one. But the point is that this is simply chain loading. It's a concept not limited to Unix. execve() does it at the level of processes, where one program can chain to another one, both running in a single process. But it is most definitely not magic, nor something entirely alien to the world of Windows.
* https://en.wikibooks.org/wiki/QBasic/Appendix#CHAIN
If you are not used to pipes on Windows, you haven't pushed Windows nearly hard enough. The complaint about Microsoft DOS was that it didn't have pipes. But Windows NT has had proper pipes all along since the early 1990s, as OS/2 did before it since 1987. Microsoft's command interpreter is capable of using them, and all of the received wisdom that people knew about pipes and Microsoft's command interpreters on DOS rather famously (it being much discussed at the time) went away with OS/2 and Windows NT.
And there are umpteen ways of improving on that, from JP Software's Take Command to various flavours of Unix-alike tools. And you should see some of the things that people do with FOR /F .
Unix was the first with the garden hosepipe metaphor, but it has been some 50 years since then. It hasn't been limited to Unix for over 30 of them. Your operating system has them, and it is very much worthwhile investigating them.
Such programs typically don’t have any interaction so the v and the e parts refer to the two ways you can control what the program does.
v: a vector (list) of input parameters (aka arguments)
e: an environment of key-value pairs
The executed program can interpret these two in any way it wishes to, though there are many conventions that are followed (and bucked by the contrarians!) like using “-“ to prefix flags/options.
This is in addition to any data sent to the program’s standard input — hence the original discussion about using pipes to send the output of one command to the input of another.
LOL. What fantasy land are you living in?
Pipes are great. Untyped, untagged pipes are not. They’re a frigging disaster in every way imaginable.
“Unix is seriously uncool with young people at the moment.”
Unix is half a century old. It’s old enough to be your Dad. Hell, it’s old enough to be your Grandad! It’s had 50 years to better itself and it hasn’t done squat.
Linux? Please. That’s just a crazy cat lady living in Unix’s trash house.
Come back when you’ve a modern successor to Plan 9 that is easy, secure, and doesn’t blow huge UI/UX chunks everywhere.
Not all of us. I much prefer the Unix way of doing things, it just makes more sense then just trying to become another Windows.
I offer you my well wishes on that.
“Linux, ew!”
A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up.
Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. Just a keyboard.
They're full of magnets, and with electron microscope you can analyze what's going on in the hardware.
- Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion)
- C/C++ are awful
- Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin? multiple users on one computer?? user groups??? entering sudo password all the time????)
- The design blunder of assuming that people will read insanely dense man pages (the amount of StackOverflow questions[0][1][2] that exist for anything that can be located in documentation are a testament to this)
- And of course no bideo gambes and other things (although hopefully webapps will liberate us from OS dependence soon[3][4])
0: https://stackoverflow.com/questions/3689838/whats-the-differ...
1: https://stackoverflow.com/questions/3528245/whats-the-differ...
2: https://stackoverflow.com/questions/3639342/whats-the-differ...
- By mixing C and C++ you look like a clueless youngster.
- /usr/local is to handle non-base/non-packages stuff so you don't trash your system. Under OpenBSD, /usr/local is for packages, everything else should be under /opt or maybe ~/src.
- OpenBSD's man pages slap hard any outdated StackOverflow crap. Have fun with a 3 yo unusable answer. Anything NON-docummented on -release- gets wrong fast. Have a look on Linux HOWTO's. That's the future of the StackOverflow usefulness over the years. Near none. Because half of the answers will not apply.
Or for that matter the likes from Apple, Google and Microsoft?
The last is convoluted crap, full of redundant functions.
Further C standards do exist, but C99 and heck, even ANSI C can be good enough for daily uses.
Perl, Ruby and PHP look close to each other, being the first the source of inspiration.
But don't try to write PHP/Ruby code as if it was something close to Perl. The same with C++.
In fact, I think Tcl would make an excellent sh replacement.
Last I checked this is something all the mainstream desktop operating systems support, so for you to call this out as a linux weirdness is baffling.
It has its pain points, but the things that (ba)sh is good at, it's really good at and python, in my experience, doesn't compete. Dumb example: `tar czf shorttexts.tar.gz $(find . -type f | grep ^./...\.txt) && rsync shorttexts.tar.gz laptop:`. I could probably make a python script that did the equivalent, but it would not be a one-liner, and my feeling is that it would be a lot uglier.
- Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin? multiple users on one computer?? user groups??? entering sudo password all the time????)
In order: /usr/bin is for packaged software; /usr/local is the sysadmin's local builds. Our servers at work have a great many users, and we're quite happy with it. I... have no idea why you would object to groups. If you're entering your sudo password all the time, you're probably doing something wrong, but you're welcome to tell it to not require a password or increase the time before reprompting.
- The design blunder of assuming that people will read insanely dense man pages (the amount of StackOverflow questions[0][1][2] that exist for anything that can be located in documentation are a testament to this)
I'll concede that many GNU/Linux manpages are a bit on the long side (hence bro and tldr pages), but having an actual manual, and having it locally (works offline) is quite nice. Besides which, you can usually just search through it and find what you want.
- And of course no bideo gambes and other things (although hopefully webapps will liberate us from OS dependence soon[3][4])
Webapps have certainly helped the application situation, but Steam and such have also gotten way better; it's moved from "barely any video games on Linux" to "a middling amount of games on Linux".
You know that no one uses Bash these days right?
Fish shell, Zsh are the much newer and friendlier shells these days. All my shells run on Fish these days.
I have to tell you that my shell ergonomics has improved a lot after I started using fish shell.
But to your point, _why_ is Bash awful?
I strongly disagree although, just like you, I have no evidence to support my claim
Haha. Fair enough!
https://trends.google.com/trends/explore?date=all&q=bash,awk...
> everything is text
I'd often like to send something structured between processes without needing both sides to have to roll their own de/serialization of the domain types; in practice I end up using sockets + some thrown-together HTTP+JSON or TCP+JSON thing instead of pipes for any data that's not CSV-friendly
> everything (ish) is a file
> including pipes and fds
To my mind, this is much less elegant when most of these things don't support seeking, and fnctls have to exist.
It'd be nicer if there were something to declare interfaces like Varlink [0, 1], and a shell that allowed composing pipelines out of them nicely.
> every piece of software is accessible as a file, invoked at the command line
> ...with local arguments
> ...and persistent global in the environment
Sure, mostly fine; serialization still a wart for arguments + env vars, but one I run into much less
> and common signalling facility
As in Unix signals? tbh those seem ugly too; sigwinch, sighup, etc ought to be connected to stdin in some way; it'd be nice if there were a more general way to send arbitrary data to processes as an event
> also nice if some files have magic properties like /dev/random or /proc or /dev/null
userspace programs can't really extend these though, unless they expose FUSE filesystems, which sounds terrible and nobody does
also, this results in things like [2]...
> every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors
iow, things get trickier once I need more than one input one one output. :)
I also prefer something like syslog over stderr, but again this is an argument for more structured things.
[0]: https://varlink.org/
[1]: ideally with sum type support though, and maybe full dependent types like https://dhall-lang.org/
[2]: https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
Well, as the process can arbitrarily change which stdin it is connected to, as stdin is a file, you need some way to still issue directions to that process.
However, for the general case, your terminal probably supports sending sigint via ctrl + c, siquit via ctrl + backslash, and the suspend/unsuspend signals via ctrl + z or y. Some terminals may allow you to extend those to the various other signals you'd like to send. (But these signals are being handled by the shell - not by stdin).
Other signals would be provided by other resources, ofc; e.g. sigint should probably be present in all programs.
---
In general, my ideal OS would have look something like [0], but with isolation between the objects (since they're now processes), provisions for state changes (so probably interfaces would look more like session types), and with a type system that supports algebraic data types.
[0]: https://www.tedinski.com/2018/02/20/an-oo-language-without-i...
you can pass as many FDs to a child as you want, 0/1/2 is just a convention
(1) everything is text
(2) everything (ish) is a file
(3) including pipes and fds
(4) every piece of software is accessible as a file, invoked at the command line
(5) ...with local arguments
(6) ...and persistent globals in the environment
A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary. It just helps.
Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material.