News for hackers. Stuff that matters.
Re:Shell Script Best Practices, from a decade of scripting t... by ducktective on Thursday October 27, 2022 @08:23AM (#33356477)
>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against.

Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data.

dmenu + fzf + jq + curl is my bread and butter in shell scripts.

However, I still haven't managed to find a way to do a bunch of tasks concurrently. No, xargs and parallel don't cut it. Just give me an opinionated way to do this that is easily inspectable, loggable and debuggable. Currenly I hack together functions in a `((job_i++ < max_jobs)) || wait -n` spaghetti.

Re:Shell Script Best Practices, from a decade of scripting t... by kqr on Thursday October 27, 2022 @08:30AM (#33356524)
I think this comment points to an even deeper insight: shell is a crappy programming language but with amazing extensibility.

I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell.

The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need.

And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.

Re:Shell Script Best Practices, from a decade of scripting t... by chasil on Thursday October 27, 2022 @08:47AM (#33356687)
The shell is an ambiguous language that cannot be directly implemented with an LR parser.

Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement.

This presentation sums up the woes of an implementor:

https://archive.fosdem.org/2018/schedule/event/code_parsing_...

Re:Shell Script Best Practices, from a decade of scripting t... by synergy20 on Thursday October 27, 2022 @09:11AM (#33356960)
new to 'rq', it's not in active development, any other alternatives? it seems doing a lot other than convert structured data to json.
Re:Shell Script Best Practices, from a decade of scripting t... by ducktective on Thursday October 27, 2022 @09:39AM (#33357310)
Not sure what it is doing more...I'm referring to this rq: https://github.com/dflemstr/rq#format-support-status

It converts to/from the listed formats.

There is also `jc` (written in Python) with the added benefit that it converts output of many common unix utilities to json. So you would not need to parse `ip` for example.

https://github.com/kellyjonbrazil/jc#parsers

Re:Shell Script Best Practices, from a decade of scripting t... by charsi on Friday October 28, 2022 @01:05AM (#33367883)
Also look at `yq` - https://github.com/mikefarah/yq

This is a wrapper to jq that also supports yaml and other file formats.

Re:Shell Script Best Practices, from a decade of scripting t... by maximus-decimus on Thursday October 27, 2022 @10:59AM (#33358536)
How do you use dmenu for your shell script? to launch it? to prompt the user for input while it's running?

Do you have an example of a script you wrote?

Re:Shell Script Best Practices, from a decade of scripting t... by ducktective on Thursday October 27, 2022 @11:29AM (#33359061)
Yes, for creating ad-hoc mini-UIs so the user can select an option. Same with fzf, but it's terminal-bound (rather than X-bound).

The scripts are similar to this one:

https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kil...

Re:Shell Script Best Practices, from a decade of scripting t... by maximus-decimus on Thursday October 27, 2022 @10:32PM (#33366876)
Thanks, I will definitively use that kill one.
Re:Shell Script Best Practices, from a decade of scripting t... by sargstuff on Friday October 28, 2022 @11:34AM (#33372751)
WASM gawk with html as user input/output more flexible.
Re:Shell Script Best Practices, from a decade of scripting t... by rustyminnow on Thursday October 27, 2022 @01:14PM (#33360788)
Do you have examples of concurrent use-cases that xargs and parallel don't satisfy? I discovered parallel recently and was blown away by how much it improves things. I've only really used it in basic scenarios so far, just wondering where its limitations are.
Re:Shell Script Best Practices, from a decade of scripting t... by ducktective on Thursday October 27, 2022 @02:35PM (#33361966)
running a bash function with its own private variables in parallel. without having to export it.
Re:Shell Script Best Practices, from a decade of scripting t... by pcthrowaway on Thursday October 27, 2022 @03:59PM (#33363185)
Can you give an example of how you'd use rq in this pipeline? I'm not finding any good examples
Re:Shell Script Best Practices, from a decade of scripting t... by ducktective on Saturday October 29, 2022 @02:44AM (#33381620)
curl -s "give.me.some/yaml" | rq --input-yaml --output-json | jq '.my.selected[1].field'