Part of the problem is that the output of commands is both a UI and an API, and because any text UI can be used as an API, the human readable text gets priority. Shell scripting is therefore kind of like building third party browser extensions. You look and you guess, and then you hack some parser up based on your guess, and hope for the best.

I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Then stdout can be for humans, and while you can parse it, you know what you’re doing is fragile.

Of course, that’s totally backwards incompatible, and as long as we’re unrealistically reinventing CLIs from the foundations to modernize them, I have a long list of changes I’d make.

Have you seen some of the existing projects currently working on it? Most well known is https://www.nushell.sh/, amongst some others

I really want to agree because it seems to make so much sense in theory. It gets rid of the need to parse by standardizing on syntax. But it doesn't solve everything, so we still don't get to live the dream. Namely, it does not solve the issue of schemas, versioning, and migration.

And this is a really big issue that threatens to derail the whole project. If my script runs a pipeline `a | b | c` and utility b gets updated with a breaking change to the schema, it breaks the entire script. Now I've got to go deep into the weeds to figure out what happened, and the breakage might not be visible in the human-readable output. So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout, and then sit there eyeballing the json to figure out how the schema changed and what I need to do to fix my script.

Seems like a big mess to me. Unless there's something I'm missing?

There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers.

That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine.

This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.

> So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout

I'm not sure how that's a particular burden. If you have `a | b | c` and you want to debug the output of b, you already have to pull that out and debug it or `tee` it, because its stdout is being piped otherwise. This would be the same, except that you'd pipe (or tee) "machine out" to stdout.

Or, since we're making wild backwards incompatible changes anyway, add a directive to the shell that makes it dump all "machine out" to stdout, a la "set +x". Now you don't even have to change your code. Just wrap the line in `set -o dump-machine-out` and `set +o dump-machine-out`

> I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract.

Except for that last jsonl part, various commands already do something like this with stdout by detecting what's on the other end.

https://unix.stackexchange.com/questions/515778/how-does-a-p...

Yeah, but it's kind of a mess. You have multiple options when implementing it without a clear standard, and in practice, it's a lot trickier to make sure that all of your code does this consistently than it is to simply use different file descriptors for different output types.

And then if the user wants to debug their script, they need to know to pipe your command to `cat` or whatever to see what's actually getting passed through.

Yup. It really grinds my gears that people came up with fairly decent ideas half a century ago, and a large amount of people decided to take that as gospel rather than as something to improve on.

And it's like pulling teeth to get any improvement, because the moment somebody like Lennart tries to get rid of decades of old cruft, drama erupts.

And even JSON is still not quite there. JSON is an okay-ish idea, but to do this properly what we need is a format that can expose things like datatypes. More like PowerShell. So that we can do amazing feats like treating a number like a number, and calculating differences between dates by doing $a - $b.

If you want something more complex and restrictive you could easily make it out of JSON. JSON works because it is simple and isn't being constantly distorted into something more complicated to cover niche use cases.

Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs!

I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software?

> Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss!

So does everything else, because of consumer drives. Fun fact: filesystems by default only guarantee the integrity of the filesystem's structure itself. The promise is that after power loss, the basic structures of the filesystem won't be corrupt, but makes no big claims of reliability about the data written. Blocks of random junk in your data, blocks of NULLs, even parts of other files (maybe deleted data) are all things I've seen happen.

Databases and the like take serious effort to ensure safety, but that greatly slows down performance. So it's kind of a hard sell for a log system that's not supposed to be a performance impact.

For this contingency, you can do log shipping, but in general, system logs shouldn't be expected to be reliable in the face of a crash, since to my knowledge all normal daemons (including rsyslogd) buffer data before flushing to disk.

> You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs!

No, it doesn't. There are file arguments to journalctl. Read the manpage, sheesh.

> I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc.

Meh. Paranoia.

> Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software?

Free Software is a licensing/distribution concept completely unrelated to whether one likes or not Microsoft's technical decisions. Some I really hate, and some are actually pretty cool.

I'm for Free Software because I like the licensing philosophy, not because I believe Unix is the best thing since sliced bread. In fact I believe Unix started as a bunch of good ideas but that have not kept up and so needs a bit of work to remain a good system to use.

> No, it doesn't. There are file arguments to journalctl. Read the manpage, sheesh.

... And how are you going to produce the journal file to read? ... with systemd. If you broke the system and it won't boot, you'll need to boot from systemd and check with journalctl because the journal can't be accessed otherwise. That usually requires a liveUSB running systemd to pull off. This is why you don't use binary logs.

Compared to `less /var/log/messages` from any Linux, I know which one I'm trusting.

I like controlling my system instead of having it controlled for me, thanks.

> And how are you going to produce the journal file to read?

What do you mean "produce"? They were produced while it was running, they can be found in /var/log/journal

> If you broke the system and it won't boot, you'll need to boot from systemd and check with journalctl because the journal can't be accessed otherwise.

Obviously? I'm not seeing the problem. It's not like you're getting anything from that system without having a way to mount your XFS/Ext4/BTRFS/LVM/luks/whatever setup. You need to boot a distro compatible with that to do it.

So of course you have to boot a Linux distro, which will easily have all the tooling available, including to deal with the journald stuff.

It's just a complete non-problem.

> I like controlling my system instead of having it controlled for me, thanks.

I'm not sure what that means exactly.

This style of argumentation is annoying because you're not even participating, you're looking for reasons to dismiss.

A sane system doesn't need a whole lot just to check logs. LVM and LUKS are different due to cryptographic needs. systemd meanwhile has little reason to store logs in binary format. The promises that are alleged are not concerns to anyone except enterprise.

> This style of argumentation is annoying because you're not even participating, you're looking for reasons to dismiss.

I just don't buy the problem as legitimate. It's an aesthetic problem, not a real problem. Sysadmins clearly have no problem with the fact that XFS is not a human readable format, or I don't recall anybody making a stink about using Berkeley DB for a whole bunch of stuff.

> systemd meanwhile has little reason to store logs in binary format.

Quite a few actually. Indexing, transparent compression, clear storage of arbitrary amounts of data with well delimited fields, quick seeking. Makes for a compact and very well performing system.

You can't quickly seek a .gz text file, while journald will tell you what happened a week ago at 3 AM in a few ms.

> The promises that are alleged are not concerns to anyone except enterprise.

Or people who realize there's a bit more to logs than 'tail' and 'grep'.

Eg, journald trivially will give you a log from a given timeframe that interleaves the logs of a proxy, httpd, database and application server, actually producing a log in which a request can be logically followed through the different services it went through, with timestamps in microseconds.

In an application that's designed for it, you can actually ask for logs regarding to a given host, user, etc.

If you've ever done log parsing, well, now you don't need to ever write a regex to split a .log by fields, because that was already done for you, and you can have UNIX timestamps directly instead of doing date parsing.

> In the same way programs are smart enough to know to render colored output or not, I'd love it if it could dump structured output (or not)

The even lower hanging fruit is to implement json output as a command line argument in all cli tools. I would love to see this done for the gnu core utils.

It was a really pleasant surprise to find the "-j" option to do this for the "ip" command from the iproute2 project.

AFAIK the idea is that if you need that kind of interoperability in unix/gnu, you're supposed to write your tool in C and include some libraries. Clearly not realistic in many use cases.

At least “du”, “env”, and “printenv” in Coreutils all support the “--null” option to separate output into being NUL-separated instead of being text lines.

There is always Powershell. The trouble there is that it's so rooted in .Net and objects that it's very difficult to integrate with existing native commands on any platform.

If these "native commands" had a sensible output format, integrating them with powershell would be as simple as putting ConvertFrom-Json or ConvertFrom-Csv in the middle of the pipeline. And let's be real, it's as poorly "integrated" with "native commands" as bash or zsh is poorly integrated with native commands.

You're not wrong, but the fix is then, "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. Building sensible serialized object-oriented output is the same problem as building sensible object-oriented output. That's why jc exists, I suppose.

There is always ConvertFrom-String. The problem is that it is one of the least intuitive and worst-performing commands I've used in Powershell. It's awful and I hate it. It's like writing sed and awk commands without the benefit of sed and awk's maturity and ubiquity. IMX, only Compare-Object has been worse.

> "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve.

It's pretty much what powershell did, though.

Eh, not really. Powershell itself is fairly limited in terms of functionality. It's basically one step removed from a .Net REPL, and the .Net classes aren't necessarily written to do *nix admin tasks. It gives you all of .Net to build tools with, but sometimes you still run into Microsoftisms that are total nonsense. There's a reason every C# project was using Newtonsoft's JSON.Net instead of the wonky custom data representations that MS was trying to push left over from the embrace/extend/extinguish era.

I see a lot of buzzwords and attacks in this comment but nothing actually concrete.

If you think .Net is the best way to interact with a *nix environment for administration, by all means go ahead. I think you'll have a lot of fun discovering the problems with it. Lie most tasks involve a pipeline of native commands. Well, if you want a Powershell command in the middle, that means you have to run a native command, marshall the output to an object, do the Powershell commands, then serialize that output back into characters, and then output to a native command. And you potentially have to deal with Powershell thinking in UTF-16 and *nix thinking in UTF-8.

However this is social media, not an academic paper, and you didn't put much effort into backing your claims, either.

It sounds like there needs to be a monolithic solution built from the ground up for interoperation.

Updating coreutils is a losing game at this point. Of course people will get angry when a well designed solution gradually takes over simply, because it is better.

What would I use PowerShell to do, though? Most of Windows is a GUI.

That's a key problem with the developer story on Windows, especially coming from GNU/Linux. Where are my standard compilers and libraries? What use do I have for a terminal on Windows? In bash, I get powerful commands and a simple text-based pipeline hooked up to coreutils and literally anything else that runs on the command line, which is a metric ton of software in that ecosystem.

Back to Windows. What would I want PowerShell to do? What would I use PowerShell for that I wouldn't want to just use Bash instead? Windows doesn't have coreutils or nice command line software to do fun or powerful things with.

I see PowerShell as a "look we have a terminal!" from Windows, but nothing that I want to do in said terminal to motivate me to learn.

> Windows doesn't have coreutils or nice command line software to do fun or powerful things with.

Sounds like you've never actually used windows for real tech stuff...

What 'real tech stuff'? Windows is an OS for consumers, not makers.

> Trying to even parse an nginx log file can be annoying with just awk or some such.

You probably already know this, but for those who do not, you can configure nginx to generate JSON log output.

Quite handy if you are aggregating structured logs across your stack.

When it comes to parsing server logs, it's too bad the functionality can't be extracted out of something like logstash, since that's already basically doing the same thing.

Though I guess the real endgame here is for upstream tools to eventually recognize the value and learn how to directly supply structured output.

You can get that out of journald.

    journalctl -o json

And applications using journald directly can provide their own custom fields.

The problem is that nobody has built an actual ffi solution except maybe the GObject guys. C isn't an ffi, because you need a C compiler to make it work. By that I mean it is not an interface, but rather just C code whose calling part has been embedded into your application.