If I had to decide on a tool to replace shell script I would vote for tcl, since it maintains many of the advantages of the shell but provides a better handling of the programming aspect. Unfortunately these days it seems that you either use shell or some full-scale language like python.
Disclaimer: I am the author of the article and JC.
It's like I get it, the old way is ugly and not always easy to decipher but at least it's shorter and your chances of understanding it are better.
I've had both -- the classic piped chain of UNIX commands and various JSON-producing tools piped to `jq`. The former were still easier to work with.
https://ilya-sher.org/2018/09/10/jq-is-a-symptom/
My bias: I'm the author of Next Generation Shell.
It's syntax isn't 100% POSIX compatible so there is some new stuff to learn but it works with all the existing POSIX tools and is more readable than AWK and Perl but while also being terse enough to write one liners.
I considered python but Ruby’s easier chaining with map/filter/etc made it easier for me to use when writing just one line with it to transform some json.
https://github.com/kellyjonbrazil/jello
Other languages are superior in their handling of maps/arrays, but Python is just so damned popular now I thought it was a good choice to democratize JSON handling.
https://blog.kellybrazil.com/2020/03/25/jello-the-jq-alterna...
What I like to do is comments like:
/*
* Collates
* [
* {
* id: 4
* dept: 'oncology',
* name: 'Joe S.'
* }
* .
* .
* .
* ]
*
* into
*
* {
* 4: { // id
* 'Joe. S': { // name
* dept: oncology
* }
* .
* .
* .
* }
*/
( Then insert horrible one-liner that does the transformation. ) /*
* Collates
* [
...
Removes all the unneeded vertical space created by inserting the extra newlines and makes the entire thing more readable. Put two spaces in front of each line that will be part of the code block and then any extra needed spaces. Some text editors can format text in this manner automatically for you.The Unix command line is the Candy Crush of interfaces, giving us a little dopamine hit every time we solve a problem that didn't need to be solved.
More long term, I have high hopes of having an environment above the kernel that basically does what /proc and /sys and others do currently. Programs equivalent to those in /bin /usr/bin etc. would be just dynamically loadable modules or built into the basic tool set. No shell scripts anymore, no random filesystems with custom formats for everything, just a JITed VM that has a strong set of tools but anybody can extend it with either stuff written in some Clojure-like language, or something compiling to the VM or something native that has some kind of interface (FFI?) to be usable from the programs running on the VM (e.g. for cryptographic stuff or stuff that needs to be as efficient as possible). We would also need something like SSH but for structured data that would support a SHELL/ REPL-like workflow as a byproduct but really be meant for more or less high performance, efficient communication (e.g. useable even for large file copy operations and such). In the end, parts of this system could connect to a in kernel VM (BPF?) and execute there but we would interact with them using the nice, structured REPL.
This would be a huge undertaking but I can't really see, how we can radically improve the efficiency of work with the current systems. It seems, we are mostly just patching old approaches to do new tricks and to me, it seems to be falling apart. The complexity we impose upon us is crushing and I don't think all of it is necessary.
Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If lines are too long, I can chop them down to a reasonable size on some sentinel.
IMO, JSON, like Javascript, is web/browser centric. For someone who rarely uses a browser or Javascript and is comfortable with UNIX, e.g., yours truly, JSON is not particularly advantageous. For large data, line-based is more robust (and memory-efficient) than JSON, IME.
Better than JSON is netstrings or bencode.
No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces of data that are nested more than one level down.
The lack of length prefixes in JSON does indeed make a solid parser somewhat more difficult, but you get the ability to author and validate it manually instead. All in all a draw and not because of the incremental parsing thing.
(Tabular or otherwise homogeneous data is indeed reprsented wastefully, but unless the individual records are huge json+gzip is a perfectly serviceable “worse-is-better” solution, and its self-describing nature along with support for structured cell values can at times make for a better experience than TSV. And at other times not.)
[1] https://github.com/ICRAR/ijson
It comes with an example program called json_reformat which I have experimented with in the past.
However the "reformatters" I write using only shell utilities work just as well. YMMV.
I doubt your JSON reformatter was entirely correct (hey, we all must parse [X]HTML using regex from time to time), as you either have to pretend you can make sense of JSON using regular expressions (you can’t, no language or format supporting unbounded nesting can be regular) or write what’s essentially a standard JSON parser in shell with all the associated inefficiencies. My own needs are usually adequately served by the likes of jq -r ... | while IFS=$"\t" read -r ..., which feels much less hackish.
(Even correctly handling CSV with quoting, escaping, and embedded newlines in a UNIX pipeline without a purpose-built utility is surprisingly difficult—although not impossible, as CSV is a regular language.)
Not everyone suffers from that fetish, fortunately. With excessively complex JSON, I prefer to extract the specific data I want. The speed of json_reformat demonstrates how slow reformatting is when done "correctly".
Every user's needs are different. jq fails to meet mine.
It doesn't. I have personally written JSON parsing code that runs on embedded systems with less RAM than the size of the file.
I have too. I used shell utilities. :)
YAJL will also work.
It also was unbearably slow in a modern computer, to the point that it made debugging difficult.
Unix shells, for all their idiosynchronacies, tend to focus on small bits of functionality directly composed from common shell or system utilities. The complexities lie in outdated design, not in the language or form.
We could definitely do with an upgrade to the underlying tooling, I don't find anything otherwise lacking in shell scripting otherwise. I don't agree with the OP that we need a web readable data format - perhaps the jq syntax could be useful but the best part of shell scripting is it is approachable by humans - if it is not you are doing too much and should be writing your missile control program in python or c, or similar. I dont think it should be easier to produce functioning junk nor should it be a requirement to master a data syntax in order to print a formated date. Maybe you need a different date program if you need to do something special.
Maybe to you pluggable means an output object from a utility being the same input object as another utility? In this case, Unix isn't pluggable at all.
It is the only shell that ships by default on modern OSes that builds up on Xerox PARC ideas.
It is not only .NET, rather anything on the OS.
It also handles COM, DLLs, OLE Automation, WMI, pluggable filesystems.
With Powershell you can easily automate something like use the currently selected cell on an Excel document, and use it as input for something else, including another active application.
Something like that would be possible on UNIX shells with DBus like protocols, and ability to load shared objects into the shell, but on typical UNIX fashion everyone does their own thing and thus the whole experience remains fossilized.
Or Perl. That's what Perl was originally created for.
Or sed. If awk is too mainstream for you.
Or Python.
Or just write the ugly pipeline in your script, and promptly forget how it actually works. I've done that a lot.
After learning how to use ed in scripts, I've found it's actually easier to use it compared to sed because it effectively has random access through the input rather than going from beginning to end.
But both awk and Perl have the advantage of storing values in variables over sed or ed.
Being able to do something like
/some string/-2 d
Which deletes a line 2 lines above where some string matched is something that's trivial to do in ed, but takes a bit of work in sed.What i would like is an cli awk ide. (Rolls of the tongue right? )
Ideally i could write ... | awk-ide "scripts/thing.awk" | ...
If it exists, run it. Otherwise have it open an editor session that runs an awk program as i type it on some buffered data, shows the result, and have some hints on what syntax/variables are available.
while inotifywait --event modify,move_self,delete_self script.awk; do awk -f script.awk input.txt; done
in a tmux pane on the right with vim/emacs/whatever on the left?Note: You need to watch the move_self and delete_self events, since editors might use some file move/deletion dance instead of a direct modify to help prevent data loss in the event of a crash.
The industry has approximated this state of affairs by approximating a GNU+Linux monoculture. Doesn’t matter that BSD tar behaves differently if ~nobody uses it.
Plus there are a couple of POSIX like OSes for embedded deployment.
Unix Text Processing, Dougherty and O'Reilly, 1987 https://www.oreilly.com/openbook/utp/
Cherry pick chapters like those on the Shell and AWK, and avoid those on troff, macros etc unless specifically interested.
Having JSON as an intermediate representation to work on instead is not going to save anyone - what we'd really need is for all tool versions/variants output on all platforms (all GNU/Linux distros, all the BSDs, all embedded Linux variants, all commercial UNICES, etc.) to be the same, all the time. That's not going to happen, so shell scripting is going to stay messy.
Also, for my INTERACTIVE shell, anyone can try to pry free-form, text-based and semi-structured output from my cold, dead hands. JSON or YAML output might be an acceptable compromise between being easy to parse and bearable for human consumption, but for my daily work, I would rather have my tools make it easy for me, the human part in the whole equation, and not some parsing logic that might not even (need to) exist. Shell scripting provides most of its value from the fact that since I'm in its repl-of-sorts all the time, I can translate that familiarity to scripts and executables effortlessly, and I would not want that going away. But I am rather certain it would, if we had JSON (or another form of more structured data interchange syntax) adopted as the "universal" interface between UNIX tools.