You'd standardize it once via an RFC and you'd be done with it.
I don't think your problem is as big as you say it is.
The real problem is that this pile of code we already have kind of works and it's already making trillions for its users. Changing the whole ecosystems would cost millions and millions, for only a very long term and unclear benefit.
In other words, worse is better.
As oblio said, you can come up with a standard conversion of structured data to text. The other way round you need write a parser for every textual output format, and typically people come up with fragile ad-hoc parsers that don't deal with edge cases properly.
Of course, who am I kidding, in real life we have some sort of crappy text interface which is half-baked both for humans and for machines. But we've been using it for almost half a century and it's too widespread to redo, so there we are, plowing through it daily.
1) We don't need to depend on each individual program's programmer to present every control and information consistently between the 2 interfaces.
2) Automation matches normal, manual use. Just put what you normally do on the command line in a file and you're done. There's no need to look through documentation on how to do what you so frequently do, only in a manner that you rarely do.
Also there is no need really to change anything (as in, breaking existing scripts). Selecting a different output format can simply be a command line option. Many tools already offer Json, XML or CSV output. But since development of those tools is so decentralized you'd be hard pressed getting them all to agree on one. But theoretically you can pick any tool you want right now, add --json support and submit a patch.
Are TUIs like htop, tmux, vim, emacs, less, etc. going to be impossible now, or will you do the negotiation protocol? Both options suck.
When programs have both normal output and errors intermixed are the objects going to be intermixed in the output that's presented to the user? For example, if you do a `find /etc/pacman.d`, instead of:
/etc/pacman.d
/etc/pacman.d/mirrorlist.pacnew
/etc/pacman.d/gnupg
/etc/pacman.d/gnupg/trustdb.gpg
/etc/pacman.d/gnupg/crls.d
find: ‘/etc/pacman.d/gnupg/crls.d’: Permission denied
/etc/pacman.d/gnupg/.gpg-v21-migrated
/etc/pacman.d/gnupg/private-keys-v1.d
find: ‘/etc/pacman.d/gnupg/private-keys-v1.d’: Permission denied
/etc/pacman.d/gnupg/tofu.db
/etc/pacman.d/gnupg/openpgp-revocs.d
find: ‘/etc/pacman.d/gnupg/openpgp-revocs.d’: Permission denied
/etc/pacman.d/gnupg/gpg.conf
/etc/pacman.d/gnupg/secring.gpg
/etc/pacman.d/gnupg/pubring.gpg~
/etc/pacman.d/gnupg/pubring.gpg
/etc/pacman.d/mirrorlist
[
"/etc/pacman.d",
"/etc/pacman.d/mirrorlist.pacnew",
"/etc/pacman.d/gnupg",
"/etc/pacman.d/gnupg/trustdb.gpg",
"/etc/pacman.d/gnupg/crls.d",
[
{
type: "Permission denied",
path: "/etc/pacman.d/gnupg/crls.d"
},
"/etc/pacman.d/gnupg/.gpg-v21-migrated",
"/etc/pacman.d/gnupg/private-keys-v1.d",
{
type: "Permission denied",
path: "/etc/pacman.d/gnupg/private-keys-v1.d"
},
"/etc/pacman.d/gnupg/tofu.db",
"/etc/pacman.d/gnupg/openpgp-revocs.d",
{
type: "Permission denied",
path: "/etc/pacman.d/gnupg/openpgp-revocs.d"
},
"/etc/pacman.d/gnupg/gpg.conf",
"/etc/pacman.d/gnupg/secring.gpg",
"/etc/pacman.d/gnupg/pubring.gpg~",
"/etc/pacman.d/gnupg/pubring.gpg",
"/etc/pacman.d/mirrorlist",
]
]
You could have every function incorporate their errors into their normal output, but that means giving up a standardized way of working with errors and warnings. I don't know if you know this, but when you do substitution or piping, by default, only stdout is used. That means that we you do piping, the programs in the pipelines normally do not see the errors in their inputs, and the errors of the multiple concurrently running programs are shown to you intermixed while the pipeline is working. That's a friggin' incredible effect that came from simple design, but when each one would output objects, you'll could get syntax errors or a completely different object like what happened in the above example. You could say, "well, only make stdout an object and let stderr be text," but the fact that they're both the same type means that you can work with the error or only with your errors in pipelines and other shell constructions. For example, `find /etc 2>&1 >/dev/null` will output the directories in /etc you can't read for whatever reason. You might want to pipe that to `xargs chmod` (for whatever reason) after preparing the output to only include the paths.
Right now, programs can strike a good balance in presenting its output in a format that is both readable to humans and other programs. By forcing their output to be structured as objects, and not giving them the option of presenting 2 formats (because we don't want that either), you're removing their ability to present the output in a manner that is readable to humans.
Take for example, rspec's output (a unit testing framework):
$ rspec spec/calculator_spec.rb
F
Failures:
1) Calculator#add returns the sum of its arguments
Failure/Error: expect(Calculator.new.add(1, 2)).to eq(3)
expected: 3
got: nil
(compared using ==)
# ./spec/calcalator_spec.rb:6:in `block (3 levels) in <top (required)>'
Finished in 0.00131 seconds (files took 0.10968 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./spec/calcalator_spec.rb:5 # Calculator#add returns the sum of its arguments
The main thing is, though, that, in the question of what the universal format of communication between programs written in different languages should be, text is the simpler, more natural choice over objects. Take note, I don't mean easier. The fact that it's easier is merely coincidence. Simplicity leads to good design because it means less arbitrary choices to make. Less controversial choices to make. Choosing objects leads to more questions: What should the primary types be? Should arrays/lists allow multiple types of elements? Floating types or decimals? Precision restriction on the decimals? Should integers and numbers that allow fractional parts be the same type or different? Should we have a null type? Should we have a date primary type? What about a time primary type? What about a datetime primary type? Whatever answers you give, there will always be groups of people that will dislike them. When you chose text, the only question is really, what encoding? Utf-8. done. Natural, simple design is what we want to be the foundation that myriads of programs and languages can base themselves on and depend on.
There's only one way I'd agree with you that structured output would be nice, and that's with mono-language OSes, like a lisp OS or some other OS where all code is in the same language, and there would be no concept of programs or shared / dynamically loaded libraries or such. In an OS like that, every function is a program, and your shell is the language's REPL. This is bliss when the OS is done in your favorite language. The problem with these kinds of OSes is that we don't all like the same languages and so it'd lead to ridiculous situations where we'd translate a new language into the high level language of the OS. That's what we do in the OS known as the web browser and why we're coming up with WebAssembly.
In conclusion, multi-language OSes like those that are Unix based are awesome, and text as the basis of communication in multi-language OSes is awesome. Therefore, text as the basis of communication in Unix is awesome. :)
If you think something like this is easy much less "you standardize it once and you're done", then you are only cheating yourself out of an essential life lesson.
I’d like to see something like cvs used more, where it can handle the edge cases without breaking, but still doesn’t need a translation step
If programs used typed data, they'd still need the option to output text to present results in a format the user can understand. To do this, a negotiation protocol could be established, like skissane said. This, in my opinion, is BAD, because then there's the possibility or probability that they'll be differences in the information conveyed in the different formats.
I believe that the use of plain text as the universal format of communication between programs is one of the greatest design decisions for Unix and CLI.