Bingo.

There are at least a dozen tools and languages and syntaxes that I've used sporadically over the years - awk, sed, bash, Mongo, perl, etc. I don't use them often enough to remember exactly how they work, and so I always have to spend a few hours reviewing manuals or old code repos or an O'Reilly book.

But if I do end up using it for a few days in a row, it starts to make sense, and I improve each time I use it.

But not with jq.

It just does not make sense to my brain, no matter how many times I've had to use it. Every single time I need to use it, it requires finding some Stack Exchange or blog and just copying and pasting. Even after seeing the solution, rarely do I then really understand why or how it works. Nor can I often take that knowledge and apply it to similar problems.

About the only other syntax or language that gives me such problems is Elastic Search DSL.

Same for me... everytime I have to lookup the basics... and I love awk,perl and xpath/xslt.

I wonder if someone tried to use plain JS as a filtering language? It would be more verbose but it would be easy to remember. For example:

   [1,2,3] | js "out = 0; for (const n of this) out += n"

That would print "6". `out` would be a special variable you write to to print the result, and `this` would be the input.

Not quite that, but ramda-cli[1] which I've created solves this problem, at least for me, by offering the familiar set of functions from Ramda, and you can create pipelines with those to do operations on your data.

[1]: https://github.com/raine/ramda-cli

I've used trentm's json (formerly known as jsontool) package from npm as my default tool for command-line manipulation of JSON for many years now. It provides CLI arguments for passing JavaScript code for filtering and executing on input. I have resisted investing the time into becoming fluent in jq because I've found that many of the common use cases I have are readily handled by jsontool.

https://www.npmjs.com/package/json

Edit: added more information

A few of the tools listed here seem to work like that, or roughly similar: https://ilya-sher.org/2018/04/10/list-of-json-tools-for-comm...

I didn't check any of them out though.

My hope was to one day add JS eval support to https://github.com/SuperpowersCorp/refactorio but as you can tell by the timestamps I haven't found any time to work on it in the last 4 years.

That's a really interesting suggestion, similar to how AWK uses $0, $1 etc.

Interesting, for me it's the exact opposite.

I've tried a couple of times to get into awk, but still find the syntax arcane.

I don't know; I wouldn't presume to tell you what you do or don't find arcane, but once I understood the somewhat unusual flow of awk ("for every line, check if the line matches this condition, and if it does run this block of code") I found it's quite easy to work with. It's "arcane" in the sense that it has an implicit loop and that it's a specialized language for a very limited class of problems, but I found that for this limited class of problem it's surprisingly effective.

  > an implicit loop

As an occasional awk user, I'd love if you expand on this. Maybe it will help clear things up for me. You're not referring to the fact that awk operates on every line independently, are you?

My mental image of awk has always been something along these lines:

    for line in readfile()
        for block in script:
            if block.match(line)
                run_block(block)
            end
        endfor
    endfor

Where the "for line in readfile()" is the "implicit loop", and the blocks are the "condition { .. }" blocks.

The actual flow is a little bit more complex and has some exceptions e.g. (BEGIN/END), but this is about the gist of it.

Thanks. Yes, I agree that my mental image is pretty much the same but it's nice to see it expressed in Python modulo end keywords ))

To expand on the other reply, there are a couple more implicit loops. There's a loop over all of the command line arguments/files, then a loop for every line in each of those files, then there is kind of a loop over the whitespace delimited fields of each of those lines. The main thing that helped me understand AWK was that every block in a script is just a pattern/action pair. When I saw snippets like

  ... | awk '{print $2}' 

I thought there was all this confusing syntax, but something like

  awk '/pattern/ {print}'

was more clear to me. In the first case, the empty pattern matches every line of the input, and the action is simply to print the second field of each line. Patterns can vary in complexity from the empty pattern to long chains of logical operators and regular expressions, such as /pattern/ in the second example. The outer quotes are just to prevent the shell from eating your dollar signs or other special characters. In a standalone AWK script you can write it like

  /pattern/ {
    print
  }

which also makes it look more like another language.

If you can get your hands on a copy of The AWK Programming Language, it's a pretty quick and pleasant read that helped everything make more sense to me. I do most of my data analysis for my research using AWK and really enjoy working with it.

  > The AWK Programming Language

I see it's public domain and discussed here on HN: https://news.ycombinator.com/item?id=13451454

I'll go over it, thank you very much for the suggestion.

Same issue. However, I do successfully rely on using ctrl-r a lot to search prior invoked commands. And have a few core aliases that I've cobbled together....

Here because.... I didn't know of ctrl-R. What a life changer (although I had an alias for "hg" to "history | grep" :) )

Please check FZF [1] and it’s integration with ctrl-r. It’s a huge productivity boost and I cannot live without it.

[1] https://github.com/junegunn/fzf

I think I actually HAVE fzf installed, thus my first experience w/ Ctrl-R was even better! I really haven't fully grok'ed fzf yet though, probably should look up some guides.

There's also McFly[1] that does interactive history search. [1]: https://github.com/cantino/mcfly

This sounds quite awesome, it's a pity that the realization is quite 'alien' to unix way of things. IMO shell commands history should be a database by default, not just a single file that gets auto-appended with both awesome commands and crap you don't really want to get stored for later.

Thanks for linking this project, will try it, it may be a game changer.

I use both awk and jq infrequently enough that I tend to struggle with anything non-trivial. I think zq would be the same.

> Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forgetting it.

I think this is the main thing. I’d prefer a streamlined CLI tool where you passed in some JS code and it’d just run it on the input (with the same slurp/raw args as jq). Could just be npm with underscore.js autoimported.

This is ironic - I use `awk` so infrequently, I have no idea how to use it without reading its man page or using Google. But I use `jq` often and find it simple.