News for hackers. Stuff that matters.
Re:Bringing the Unix philosophy to the 21st century (2019) by c0l0 on Sunday August 22, 2021 @01:05PM (#28267267)
So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise.

Having JSON as an intermediate representation to work on instead is not going to save anyone - what we'd really need is for all tool versions/variants output on all platforms (all GNU/Linux distros, all the BSDs, all embedded Linux variants, all commercial UNICES, etc.) to be the same, all the time. That's not going to happen, so shell scripting is going to stay messy.

Also, for my INTERACTIVE shell, anyone can try to pry free-form, text-based and semi-structured output from my cold, dead hands. JSON or YAML output might be an acceptable compromise between being easy to parse and bearable for human consumption, but for my daily work, I would rather have my tools make it easy for me, the human part in the whole equation, and not some parsing logic that might not even (need to) exist. Shell scripting provides most of its value from the fact that since I'm in its repl-of-sorts all the time, I can translate that familiarity to scripts and executables effortlessly, and I would not want that going away. But I am rather certain it would, if we had JSON (or another form of more structured data interchange syntax) adopted as the "universal" interface between UNIX tools.

Re:Bringing the Unix philosophy to the 21st century (2019) by coliveira on Sunday August 22, 2021 @05:30PM (#28269817)
You mentioned one pain point of shell scripting, variability among platforms. The other pain point is that different tools have different methods to treat errors and warnings. This makes debugging shell scripts a nightmare, compare to scripting languages like python. If you need to debug your work, you need to check several levels of script vs. commands, and use different ways to check for errors depending on what is causing the problem.

If I had to decide on a tool to replace shell script I would vote for tcl, since it maintains many of the advantages of the shell but provides a better handling of the programming aspect. Unfortunately these days it seems that you either use shell or some full-scale language like python.

Re:Bringing the Unix philosophy to the 21st century (2019) by pjmlp on Monday August 23, 2021 @07:19AM (#28274275)
Tcl is also a full scale language, the startup I was on during the com wave was shipping products written in Tcl + C, just like Python.
Re:Bringing the Unix philosophy to the 21st century (2019) by kbrazil on Sunday August 22, 2021 @01:21PM (#28267458)
I don’t think the issue is that it’s hard to manually parse. The problem is that it’s hard for someone else to read your ad-hoc parser years later and reason about what you did if they need to modify it.

Disclaimer: I am the author of the article and JC.

Re:Bringing the Unix philosophy to the 21st century (2019) by pdimitar on Sunday August 22, 2021 @01:41PM (#28267656)
This is even more true for the ungodly long `jq` incantations that people write.

It's like I get it, the old way is ugly and not always easy to decipher but at least it's shorter and your chances of understanding it are better.

I've had both -- the classic piped chain of UNIX commands and various JSON-producing tools piped to `jq`. The former were still easier to work with.

Re:Bringing the Unix philosophy to the 21st century (2019) by ilyash on Monday August 23, 2021 @12:26AM (#28272333)
Opinion: Today's shells should have structured data; `jq` is a symptom of a shell that can't handle structured data.

https://ilya-sher.org/2018/09/10/jq-is-a-symptom/

My bias: I'm the author of Next Generation Shell.

Re:Bringing the Unix philosophy to the 21st century (2019) by kbrazil on Sunday August 22, 2021 @01:51PM (#28267762)
Yes, I have seen those too! That’s why I also wrote Jello, which is like jq but uses pure python without the boilerplate. Python is nearly universal now and typically easy to read, though more verbose. Jq is just as much a write-once tool as awk and perl for more complex queries. For simple attribute calls, though, it’s both terse and readable.
Re:Bringing the Unix philosophy to the 21st century (2019) by hnlmorg on Sunday August 22, 2021 @05:16PM (#28269701)
This is why I wrote murex shell (https://github.com/lmorg/murex), it's an alternative $SHELL, so you'd use it in place of Bash or Zsh, but it's optimised for modern DevOps tools. Which means JSON and YAML are first class citizens.

It's syntax isn't 100% POSIX compatible so there is some new stuff to learn but it works with all the existing POSIX tools and is more readable than AWK and Perl but while also being terse enough to write one liners.

Re:Bringing the Unix philosophy to the 21st century (2019) by NegativeLatency on Sunday August 22, 2021 @04:19PM (#28269201)
I wrote something similar by description in ruby, I’d be curious to see your python implementation.

I considered python but Ruby’s easier chaining with map/filter/etc made it easier for me to use when writing just one line with it to transform some json.

https://github.com/nburns/utilities/blob/master/rson

Re:Bringing the Unix philosophy to the 21st century (2019) by kbrazil on Sunday August 22, 2021 @04:28PM (#28269291)
Sure thing!

https://github.com/kellyjonbrazil/jello

Other languages are superior in their handling of maps/arrays, but Python is just so damned popular now I thought it was a good choice to democratize JSON handling.

https://blog.kellybrazil.com/2020/03/25/jello-the-jq-alterna...

Re:Bringing the Unix philosophy to the 21st century (2019) by hackerbrother on Sunday August 22, 2021 @04:04PM (#28269065)
(Sorry, this is my first time trying to do a formatted comment here.)

What I like to do is comments like:

/*

  * Collates

  * [

  *    { 

  *        id: 4

  *        dept: 'oncology',

  *        name: 'Joe S.'

  *    }

  *    .

  *    .

  *    .

  *  ]

  *

  *  into

  *

  *  {

  *     4: {  // id

  *        'Joe. S': { // name

  *           dept: oncology

  *        }

  *     .     

  *     .     

  *     .     

  *   }

  */


( Then insert horrible one-liner that does the transformation. )
Re:Bringing the Unix philosophy to the 21st century (2019) by Jtsummers on Sunday August 22, 2021 @07:10PM (#28270615)
Two spaces in front of a line will create a code block:

  /*
   * Collates
   * [
  ...
Removes all the unneeded vertical space created by inserting the extra newlines and makes the entire thing more readable. Put two spaces in front of each line that will be part of the code block and then any extra needed spaces. Some text editors can format text in this manner automatically for you.
Re:Bringing the Unix philosophy to the 21st century (2019) by rodgerd on Sunday August 22, 2021 @09:16PM (#28271365)
But it's very gratifying to the person who re-invents the wheel for the millionth time and feels good because of it.

The Unix command line is the Candy Crush of interfaces, giving us a little dopamine hit every time we solve a problem that didn't need to be solved.

Re:Bringing the Unix philosophy to the 21st century (2019) by kaliszad on Sunday August 22, 2021 @04:40PM (#28269388)
I can see the potential in Babashka and other Clojure-based systems from the approach point of view. There needs to be a simple integrated editor or much better readline implementation though that needs to work out of the box. That should improve the interactivity in a major way. We also need to have a set of very simple tools to work with files, network connections (e.g. traffic dumps, netcat) and other typical shell tasks. This isn't really as efficient when used interactively in the Clojure ecosystem or anywhere else from what I have seen besides just using e.g. ls, du, df, fdisk, tcpdump, nc ...

More long term, I have high hopes of having an environment above the kernel that basically does what /proc and /sys and others do currently. Programs equivalent to those in /bin /usr/bin etc. would be just dynamically loadable modules or built into the basic tool set. No shell scripts anymore, no random filesystems with custom formats for everything, just a JITed VM that has a strong set of tools but anybody can extend it with either stuff written in some Clojure-like language, or something compiling to the VM or something native that has some kind of interface (FFI?) to be usable from the programs running on the VM (e.g. for cryptographic stuff or stuff that needs to be as efficient as possible). We would also need something like SSH but for structured data that would support a SHELL/ REPL-like workflow as a byproduct but really be meant for more or less high performance, efficient communication (e.g. useable even for large file copy operations and such). In the end, parts of this system could connect to a in kernel VM (BPF?) and execute there but we would interact with them using the nice, structured REPL.

This would be a huge undertaking but I can't really see, how we can radically improve the efficiency of work with the current systems. It seems, we are mostly just patching old approaches to do new tricks and to me, it seems to be falling apart. The complexity we impose upon us is crushing and I don't think all of it is necessary.

Re:Bringing the Unix philosophy to the 21st century (2019) by dimitar on Sunday August 22, 2021 @05:25PM (#28269780)
Babashka is really nice in that it has json and yaml libraries built-in, so you end up using your Clojure data structures throughout the script, and yet still consuming/emitting json/yaml if you need to.
Re:Bringing the Unix philosophy to the 21st century (2019) by pjmlp on Monday August 23, 2021 @07:21AM (#28274281)
So basically a Lisp Machines REPL.
Re:Bringing the Unix philosophy to the 21st century (2019) by 1vuio0pswjnm7 on Sunday August 22, 2021 @06:20PM (#28270239)
Can JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write.

Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If lines are too long, I can chop them down to a reasonable size on some sentinel.

IMO, JSON, like Javascript, is web/browser centric. For someone who rarely uses a browser or Javascript and is comfortable with UNIX, e.g., yours truly, JSON is not particularly advantageous. For large data, line-based is more robust (and memory-efficient) than JSON, IME.

Better than JSON is netstrings or bencode.

https://en.wikipedia.org/wiki/Netstrings

https://en.wikipedia.org/wiki/Bencode

Re:Bringing the Unix philosophy to the 21st century (2019) by mananaysiempre on Sunday August 22, 2021 @07:17PM (#28270658)
> JSON’s design assumes the user can read the entire file into memory

No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces of data that are nested more than one level down.

The lack of length prefixes in JSON does indeed make a solid parser somewhat more difficult, but you get the ability to author and validate it manually instead. All in all a draw and not because of the incremental parsing thing.

(Tabular or otherwise homogeneous data is indeed reprsented wastefully, but unless the individual records are huge json+gzip is a perfectly serviceable “worse-is-better” solution, and its self-describing nature along with support for structured cell values can at times make for a better experience than TSV. And at other times not.)

[1] https://github.com/ICRAR/ijson

[2] https://github.com/AMDmi3/jsonslicer

[3] https://github.com/danielyule/naya

Re:Bringing the Unix philosophy to the 21st century (2019) by 1vuio0pswjnm7 on Sunday August 22, 2021 @07:55PM (#28270887)
All three of those Python scripts use the same library, YAJL

It comes with an example program called json_reformat which I have experimented with in the past.

However the "reformatters" I write using only shell utilities work just as well. YMMV.

Re:Bringing the Unix philosophy to the 21st century (2019) by mananaysiempre on Sunday August 22, 2021 @08:14PM (#28270992)
Yes, but raw YAJL (or its Yajl-Py binding, or other incremental parsers like jsmn) is a right pain to use while these are actually interesting in terms of API design.

I doubt your JSON reformatter was entirely correct (hey, we all must parse [X]HTML using regex from time to time), as you either have to pretend you can make sense of JSON using regular expressions (you can’t, no language or format supporting unbounded nesting can be regular) or write what’s essentially a standard JSON parser in shell with all the associated inefficiencies. My own needs are usually adequately served by the likes of jq -r ... | while IFS=$"\t" read -r ..., which feels much less hackish.

(Even correctly handling CSV with quoting, escaping, and embedded newlines in a UNIX pipeline without a purpose-built utility is surprisingly difficult—although not impossible, as CSV is a regular language.)

Re:Bringing the Unix philosophy to the 21st century (2019) by 1vuio0pswjnm7 on Sunday August 22, 2021 @08:49PM (#28271201)
Thanks for the summary of problems with JSON. For ease of use, it will not likely become a de facto interchange format in the UNIX context. However not all JSON is the same. Simple JSON is easy to parse and one does not need jq or libraries, for example JSON DoH responses. People have even tried to standardise simpler JSON, e.g., jsonl, as another reply mentions. But, as I said, the design of JSON allows for and encourages complexity. If that's what you like, go for it. An HN commenter recently identified the "complexity fetish" that many programmers have.

Not everyone suffers from that fetish, fortunately. With excessively complex JSON, I prefer to extract the specific data I want. The speed of json_reformat demonstrates how slow reformatting is when done "correctly".

Every user's needs are different. jq fails to meet mine.

Re:Bringing the Unix philosophy to the 21st century (2019) by kbrazil on Sunday August 22, 2021 @07:17PM (#28270656)
JSON Lines is often used for this type of thing.

https://jsonlines.org/

Re:Bringing the Unix philosophy to the 21st century (2019) by user-the-name on Sunday August 22, 2021 @07:53PM (#28270871)
> JSON's design assumes the user can read the entire file into memory

It doesn't. I have personally written JSON parsing code that runs on embedded systems with less RAM than the size of the file.

Re:Bringing the Unix philosophy to the 21st century (2019) by 1vuio0pswjnm7 on Sunday August 22, 2021 @08:02PM (#28270922)
"I have personally written JSON parsing code that runs on embedded systems with less RAM than the size of the file."

I have too. I used shell utilities. :)

YAJL will also work.

Re:Bringing the Unix philosophy to the 21st century (2019) by emodendroket on Sunday August 22, 2021 @06:43PM (#28270431)
PowerShell is just so much nicer to use than anything where text munging is the only way to do things, and it’s just as “pluggable” as Unix shell commands. And it can output text (or JSON or XML or YAML) or whatever you want easily by piping into relevant commands (or just not piping anywhere if you wanted text). I don’t imagine those are going anywhere anytime soon but I think it is wrong to say the system cannot be improved upon.
Re:Bringing the Unix philosophy to the 21st century (2019) by diegocg on Sunday August 22, 2021 @07:30PM (#28270747)
I had to write a powershell script and my impression was that objects are a much worse interface than the worst text manipulation tricks. Because you are passing objects around it quickly becomes really hard to understand what is going on on the code and which object interfaces the script is using, because the object lives behind the pipe so to speak. It was my impression that it is too easy to create unreadable powershell scripts

It also was unbearably slow in a modern computer, to the point that it made debugging difficult.

Re:Bringing the Unix philosophy to the 21st century (2019) by emodendroket on Sunday August 22, 2021 @08:11PM (#28270981)
I'm not sure I understand exactly what you mean. I would rather read Select { $_.Property } than some awk or sed stuff to fetch the same thing any day. I'm not sure how this is different than most other modern languages used to write scripts.
Re:Bringing the Unix philosophy to the 21st century (2019) by smaudet on Monday August 23, 2021 @03:32AM (#28273156)
While you can produce unreadable sphagetti in any language, powershell is just a poor mans Python without any real language features. I'd much rather use csharp(script) if I have a need to interface with some ms stuff (.net system libraries or whatever), powershell is just underwhelming and bloated without any real performance or productivity gains.

Unix shells, for all their idiosynchronacies, tend to focus on small bits of functionality directly composed from common shell or system utilities. The complexities lie in outdated design, not in the language or form.

We could definitely do with an upgrade to the underlying tooling, I don't find anything otherwise lacking in shell scripting otherwise. I don't agree with the OP that we need a web readable data format - perhaps the jq syntax could be useful but the best part of shell scripting is it is approachable by humans - if it is not you are doing too much and should be writing your missile control program in python or c, or similar. I dont think it should be easier to produce functioning junk nor should it be a requirement to master a data syntax in order to print a formated date. Maybe you need a different date program if you need to do something special.

Re:Bringing the Unix philosophy to the 21st century (2019) by kaba0 on Monday August 23, 2021 @01:16PM (#28279034)
Powershell is slow to start up. Do you have any benchmark to show that it actually runs slower than reparsing the same data thousands of times as text?
Re:Bringing the Unix philosophy to the 21st century (2019) by shatteredspace on Monday August 23, 2021 @01:20AM (#28272550)
I would disagree and say that while PowerShell is well...powerful, it's not as pluggable as the Unix shell. Purely due to objects vs plaintext. PowerShell works off of .NET objects thus if I want to take output from one command and send it to another I have to verify that the receiving command can in fact receive the object type I'm sending. This is where the Unix shell to me is more 'pluggable' but it makes it more muddy as well. I can pipe plain-text information to any utility but to ensure that it will do what I want and not error out I may have to slice and dice my initial output a bit.

Maybe to you pluggable means an output object from a utility being the same input object as another utility? In this case, Unix isn't pluggable at all.

Re:Bringing the Unix philosophy to the 21st century (2019) by pjmlp on Monday August 23, 2021 @07:29AM (#28274334)
Powershell is much more than that, you are only seeing the basic stuff.

It is the only shell that ships by default on modern OSes that builds up on Xerox PARC ideas.

It is not only .NET, rather anything on the OS.

It also handles COM, DLLs, OLE Automation, WMI, pluggable filesystems.

With Powershell you can easily automate something like use the currently selected cell on an Excel document, and use it as input for something else, including another active application.

Something like that would be possible on UNIX shells with DBus like protocols, and ability to load shared objects into the shell, but on typical UNIX fashion everyone does their own thing and thus the whole experience remains fossilized.

Re:Bringing the Unix philosophy to the 21st century (2019) by emodendroket on Monday August 23, 2021 @07:23AM (#28274290)
You can always transform the output by selecting particular properties, so the commands don’t have to expect exactly the shape of your first command’s output. This is similar to what you’d do in Unix to massage text output anyway.
Re:Bringing the Unix philosophy to the 21st century (2019) by JJMcJ on Sunday August 22, 2021 @04:01PM (#28269022)
> awk

Or Perl. That's what Perl was originally created for.

Or sed. If awk is too mainstream for you.

Or Python.

Or just write the ugly pipeline in your script, and promptly forget how it actually works. I've done that a lot.

Re:Bringing the Unix philosophy to the 21st century (2019) by u801e on Sunday August 22, 2021 @04:32PM (#28269317)
> Or sed

After learning how to use ed in scripts, I've found it's actually easier to use it compared to sed because it effectively has random access through the input rather than going from beginning to end.

But both awk and Perl have the advantage of storing values in variables over sed or ed.

Re:Bringing the Unix philosophy to the 21st century (2019) by Koshkin on Sunday August 22, 2021 @06:27PM (#28270305)
sed, being a stream editor, is more efficient than ed, in that it does not need to keep the entire file in memory.
Re:Bringing the Unix philosophy to the 21st century (2019) by u801e on Sunday August 22, 2021 @08:58PM (#28271251)
True, but if the file isn't that big, then the difference isn't noticable.

Being able to do something like

  /some string/-2 d
Which deletes a line 2 lines above where some string matched is something that's trivial to do in ed, but takes a bit of work in sed.
Re:Bringing the Unix philosophy to the 21st century (2019) by athrowaway3z on Sunday August 22, 2021 @03:29PM (#28268679)
I know enough awk to know it solves my problem, but use it to little to remember how, and the development experience is never very clean.

What i would like is an cli awk ide. (Rolls of the tongue right? )

Ideally i could write ... | awk-ide "scripts/thing.awk" | ...

If it exists, run it. Otherwise have it open an editor session that runs an awk program as i type it on some buffered data, shows the result, and have some hints on what syntax/variables are available.

Re:Bringing the Unix philosophy to the 21st century (2019) by cle on Sunday August 22, 2021 @04:26PM (#28269264)
I use this occasionally in Emacs with awk-mode to live edit awk programs against a data file:

https://github.com/danlamanna/live-awk-mode

Re:Bringing the Unix philosophy to the 21st century (2019) by xelxebar on Sunday August 22, 2021 @09:54PM (#28271602)
What about something like

  while inotifywait --event modify,move_self,delete_self script.awk; do awk -f script.awk input.txt; done
in a tmux pane on the right with vim/emacs/whatever on the left?

Note: You need to watch the move_self and delete_self events, since editors might use some file move/deletion dance instead of a direct modify to help prevent data loss in the event of a crash.

Re:Bringing the Unix philosophy to the 21st century (2019) by kbrazil on Sunday August 22, 2021 @10:25PM (#28271745)
I wrote something similar to this to query JSON and JSON lines with python instead of awk for text. It’s called Jellex (Jello Explorer) which is a TUI front-end to Jello. Jello is a python analog to JQ.

https://github.com/kellyjonbrazil/jellex

Re:Bringing the Unix philosophy to the 21st century (2019) by jl6 on Sunday August 22, 2021 @01:50PM (#28267747)
> what we'd really need is for all tool versions/variants output on all platforms (all GNU/Linux distros, all the BSDs, all embedded Linux variants, all commercial UNICES, etc.) to be the same, all the time

The industry has approximated this state of affairs by approximating a GNU+Linux monoculture. Doesn’t matter that BSD tar behaves differently if ~nobody uses it.

Re:Bringing the Unix philosophy to the 21st century (2019) by trasz on Monday August 23, 2021 @08:09AM (#28274621)
Nobody except OSX, which means the great majority of Unix desktops use BSD tar.
Re:Bringing the Unix philosophy to the 21st century (2019) by pjmlp on Monday August 23, 2021 @04:49PM (#28281736)
HP-UX, AIX and Solaris are still around.

Plus there are a couple of POSIX like OSes for embedded deployment.

Re:Bringing the Unix philosophy to the 21st century (2019) by 1MachineElf on Sunday August 22, 2021 @03:20PM (#28268590)
About this few weeks training you mentioned for Unix-style parsing of unstructured text - what resources do you recommend?
Re:Bringing the Unix philosophy to the 21st century (2019) by tuatoru on Sunday August 22, 2021 @05:27PM (#28269798)
The GNU Awk manual has several practical examples in it - as manuals should. (Looking at you, nearly all man pages in linux distros.)
Re:Bringing the Unix philosophy to the 21st century (2019) by yesenadam on Sunday August 22, 2021 @07:28PM (#28270727)
The original 1988 book The AWK Programming Language by A, W and K is, in my opinion, one of the finest pieces of documentation ever written, on any subject. A joy to read and be instructed by.
Re:Bringing the Unix philosophy to the 21st century (2019) by chillfox on Sunday August 22, 2021 @09:33PM (#28271491)
And $722 AUD on Amazon. Did they never make a second printing?
Re:Bringing the Unix philosophy to the 21st century (2019) by wrycoder on Sunday August 22, 2021 @10:28PM (#28271759)
Re:Bringing the Unix philosophy to the 21st century (2019) by chillfox on Sunday August 29, 2021 @05:51AM (#28345481)
Thanks
Re:Bringing the Unix philosophy to the 21st century (2019) by seewhat on Monday August 23, 2021 @10:32AM (#28276350)
I have occasionally dipped into this book for instruction and consider it a freely accessible reference:

Unix Text Processing, Dougherty and O'Reilly, 1987 https://www.oreilly.com/openbook/utp/

Cherry pick chapters like those on the Shell and AWK, and avoid those on troff, macros etc unless specifically interested.