News for hackers. Stuff that matters.
Re:Bash 5.0 released by giancarlostoro on Monday January 07, 2019 @09:38PM (#18852611)
Any recommended reading for Bash? I'm somewhat new to it and it's interesting ways of getting things done. I've used it minimally in the past, but have found myself writing a 100> LOC script, which I can't help but feel I'm likely over-complicating certain bits and pieces.
Re:Bash 5.0 released by LukeShu on Monday January 07, 2019 @10:03PM (#18852744)
I'm a fan of reading code.

I would generally describe a GNU/Linux distro as being a "giant pile of shell scripts". That's a little less true with init scripts generally now being systemd units. But that's where I'd start: Look at the code that distros write, that isn't part of some other upstream software.

- Arch Linux's `makepkg` https://git.archlinux.org/pacman.git

- Arch Linux's `mkinitcpio` https://git.archlinux.org/mkinitcpio.git/

- Arch Linux's `netctl` https://git.archlinux.org/netctl.git/

- Downstream from Arch, Parabola's "libretools" dev tools package https://git.parabola.nu/packages/libretools.git/ (disclaimer: I'm the maintainer of libretools)

Gentoo also has a lot of good shell scripting to look a, but it's mostly either POSIX shell, or targets older Bash (their guidelines http://devmanual.gentoo.org/tools-reference/bash/index.html say to avoid Bash 3 features). I tend to believe that changes made to the Bash language over the years are enhancements, and that they let you write cleaner, more robust code.

Re:Bash 5.0 released by xorcist on Tuesday January 08, 2019 @03:55AM (#18854245)
There used to be this great resource called Linux Documentation Project. It's not as active as it used to be but it produced some really book-quality documents, including the Advanced Bash Scripting Guide at https://www.tldp.org/LDP/abs/html/ .

Read it! It's great. But know that a lot of bash scripting isn't really in bash, it's really required to be proficient with grep, sed, cut, dc and a few other text processing utilities. Learn those! Then are are a few other tricks to be mindful of: be mindful of spaces (wrap your variable substitution in double quotes, mostly), be mindful of sub-shells (piping to something that sets variables can be problematic), and a few other things that can really only be learned by reading good code.

But it's also good to know when you shouldn't venture further. My rule of thumb is when I'm using arrays or hash maps, then it's a good idea to move to another language. That's probably Python nowaways. A lot of people use tons of awk or perl snippets inside their bash scripts, that can also be a sign that it's time to move the whole script over.

Re:Bash 5.0 released by posix_me_less on Tuesday January 08, 2019 @04:39AM (#18854442)
Advanced Bash Scripting Guide on TLDP is a verbose and cumbersome-to-read writeup. Better read the bash man page and then http://mywiki.wooledge.org/BashGuide and http://wiki.bash-hackers.org/start
Re:Bash 5.0 released by JeremyBanks on Monday January 07, 2019 @09:42PM (#18852641)
It can't be done. If you want to write reliable code, and actually notice all of the possible error conditions instead of silently ignoring them, your code needs to get more verbose and complicated than it would be to just use a more capable tool like Python or Node, and it still won't be as reliable.

If you have more logic than a couple of string comparisons, Bash is not the right tool for the job.

Re:Bash 5.0 released by nwatson on Monday January 07, 2019 @09:53PM (#18852696)
I recommend Greg's Bash Wiki ... https://mywiki.wooledge.org/BashGuide. See general notes, then at the bottom of the page are many links to additional considerations.

Like others say, "bash" is a hard tool to get right (and I'm not saying I do it right either, necessarily, but Greg's Wiki was real helpful!). I'm building a hybrid bash/python3 environment now (something I'll hopefully open-source at some point), and bash is just the "glue" to get things set up so most aspects of development can funnel through to python3 + other tools.

But ... things that make bash real useful:

    * it's available everywhere (even in Windows with Ubuntu-18.04/WSL subsystem)
    * it can bootstrap everything else you need
    * it can wrap, in bash functions, aliases, and "variables" (parameters), the
      real functionality you want to expose ... the
      guts can be written in python3 or other tools
Without a good bash bootstrap script you end up writing 10 pages of arcane directions for multiple platforms telling people to download 10 packages and 4 pieces of software per platform, and nobody will have consistent reproducible environments.

EDIT: I think there's a revised version of Greg's Bash Wiki in the works.

Re:Bash 5.0 released by dorfsmay on Monday January 07, 2019 @11:05PM (#18853050)
It is availbale, almost, everywhere but be careful with the version, different Linux diatros are at different versions, the last time I used OSX it was stuck on a very old version, and I expect the different BSD OSes to run fairly new versions.
Re:Bash 5.0 released by int_19h on Tuesday January 08, 2019 @12:44AM (#18853514)
BSDs don't ship Bash as part of the base system - you have to install it from packages or ports. And that one is the most recent that the maintainer bothered to package. E.g. FreeBSD is on 4.4.23 right now, which actually appears to be newer than e.g. Debian unstable.
Re:Bash 5.0 released by AnIdiotOnTheNet on Monday January 07, 2019 @10:56PM (#18853014)
Or you just bundle your application and its dependencies into a single folder for each OS and distribute it.
Re:Bash 5.0 released by platinumrad on Monday January 07, 2019 @10:25PM (#18852855)
>Python or Node

Python, sure, but replacing Bash with Node just seems like replacing a language crippled by its need to be backward compatible with (Bourne) sh with a language crippled by its need to be backward compatible with what Brandon Eich came up with in two weeks in 1995.

Re:Bash 5.0 released by JeremyBanks on Tuesday January 08, 2019 @06:23PM (#18860803)
It wouldn't be my first choice either, but modern JavaScript isn't backwards-compatible, it is at least possible to write reliable and robust code if you take some care and know what options to enable, and it's mostly consistent across platforms. Last month, I spent a week trying to handle all of the possible error conditions in an overgrown bash script. It was immeasurably worse.

Python is the more direct substitute, though: it's built-in to almost every platform, and the Python 3 stagnation has even given us a consistent version: 2.7.

Re:Bash 5.0 released by mohaine on Monday January 07, 2019 @09:48PM (#18852669)
As they say, bash is just good enough not to get replaced.
Re:Bash 5.0 released by bpchaps on Monday January 07, 2019 @10:30PM (#18852881)
Sure it can. Treat it as a functional and immutable language, and you can go pretty far with it, pretty safely.

You're forgetting that bash really just calls other code. So you can combine two language's stdin/out functionality (eg, Python or Node) to pair code together. Sure, it won't be fast, but it can do quick wonders as an ad hoc data pipeline.

Re:Bash 5.0 released by cheez on Monday January 07, 2019 @10:51PM (#18852987)
I just use set -e at the top of the script
Re:Bash 5.0 released by JeremyBanks on Monday January 07, 2019 @11:44PM (#18853213)
That's a good idea, but it's not a complete solution. See http://mywiki.wooledge.org/BashFAQ/105
Re:Bash 5.0 released by cheez on Tuesday January 08, 2019 @08:18PM (#18861481)
Nice. But I use C++ so I'm used to shooting off my own foot.
Re:Bash 5.0 released by flukus on Monday January 07, 2019 @10:05PM (#18852752)
By far my best advice is to write auto-complete scripts if possible: https://iridakos.com/tutorials/2018/03/01/bash-programmable-... . Many of mine will do things like look up database values, so "command<tab><tab><tab>" is the workflow 99% of the time.

Keep them short. There are always exceptions but the "do one thing" mantra is handy, they can always be wrapped into a more complex workflow with a bigger script. None of my frequently used ones are over 100 LoC.

Write them for you and you alone when possible, start off simple and iterate. Fight that developer urge to solve a generalized problem for everyone.

Embrace the environment and global environment variables. We're trained to avoid global variables like the plague but they are really useful in scripts. My auto complete scripts that I mentioned above, they know which database to connect to based off the environment variable and there are separate commands to switch environment.

Make sure you aren't using it where things like make would be more appropriate.

Re:Bash 5.0 released by arthulia on Monday January 07, 2019 @09:52PM (#18852687)
I recommend Greg's Wiki. It's the only resource I use. It covers common pitfalls and anti-patterns.

https://mywiki.wooledge.org/

Re:Bash 5.0 released by gravypod on Monday January 07, 2019 @09:41PM (#18852635)
I've recently picked up a subscription to Destroy All Software and he has a few really good demos of how to architect simple and powerful shell scripts using bash. I'd really recommend it for anyone who doesn't need help with the semantics and instead needs help with the organization of a script.
Re:Bash 5.0 released by giancarlostoro on Monday January 07, 2019 @09:58PM (#18852716)
I've got a subscription too, watching him go in some of the videos as he casually does Bash blew my mind, which is another reason I ask this question. I'm trying to at least watch all his videos, I intend to work through them after watching them all at least once, and then I intend to rewrite some of the solutions (the Ruby ones) in Python and Go just to be sure I understand the concepts more, and I'm not just copying and pasting.
Re:Bash 5.0 released by dvdgsng on Tuesday January 08, 2019 @05:16AM (#18854565)
* look into parameter substitution https://www.tldp.org/LDP/abs/html/parameter-substitution.htm...

* use traps http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_12_02.htm...

* read about safe ways to do things in bash https://github.com/anordal/shellharden/blob/master/how_to_do...

* this is pretty helpful writeup of general CLI usage, yet not bash specific https://github.com/jlevy/the-art-of-command-line (related HN discussion https://news.ycombinator.com/item?id=9720813)

Re:Bash 5.0 released by rb808 on Monday January 07, 2019 @11:31PM (#18853158)
From the google Shell Style Guide: "If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. "

https://google.github.io/styleguide/shell.xml

Re:Bash 5.0 released by h1d on Tuesday January 08, 2019 @01:08AM (#18853637)
LOC alone doen't mean much. I have a backup script in fish of 400+ lines of code because it tries to dump data out of different sources like MySQL, PGSQL, InfluxDB, /etc files and others. Just use what feels comfortable achieving the task.
Re:Bash 5.0 released by notabee on Tuesday January 08, 2019 @10:51AM (#18856366)
That guide is really useful for making scripts more readable.
Re:Bash 5.0 released by shpx on Monday January 07, 2019 @11:32PM (#18853162)
I recommend reading and memorizing all 30-ish of the readline shortcuts

https://tiswww.case.edu/php/chet/readline/readline.html#SEC1...

Readline is the library bash uses for editing the input line, and it has some nice movement keys. For example Alt-b moves the cursor back a word, Ctrl-u deletes to the beginning of the line, Ctrl-w removes one word behind the cursor.

They work in a bunch of other programs, like the Python interpreter's interactive mode for example.

Re:Bash 5.0 released by Tepix on Tuesday January 08, 2019 @07:37AM (#18855083)
Excellent advice! Thanks.
Re:Bash 5.0 released by saagarjha on Monday January 07, 2019 @09:42PM (#18852642)
> have found myself writing a 100> LOC script

Often, this is a good sign that you might want to switch to another scripting language.

Re:Bash 5.0 released by kawsper on Monday January 07, 2019 @10:00PM (#18852729)
That's true, so many times I have used shellcheck and started to improve a bash script, only to realize my time was better spent rewriting the script in Ruby.
Re:Bash 5.0 released by int_19h on Tuesday January 08, 2019 @12:45AM (#18853522)
Take a look at this, as well:

https://amoffat.github.io/sh/

Re:Bash 5.0 released by TheGrassyKnoll on Tuesday January 08, 2019 @07:48AM (#18855150)
Re:Bash 5.0 released by theonemind on Monday January 07, 2019 @09:55PM (#18852704)
advanced bash scripting guide: https://www.tldp.org/LDP/abs/html/

also, https://www.shellcheck.net/ not reading, but pretty neat. static code analysis for shell scripts, points out common errors.

Re:Bash 5.0 released by asicsp on Monday January 07, 2019 @11:23PM (#18853126)
I have a list of resources related to bash/cli[1]

I would highly recommend BashGuide[2] and ryanstutorials[3] as a starting point. After that, go through rest of the wooledge site for FAQs, best practices, etc

shellcheck[4] is awesome for checking your scripts for potential downfalls and issues

[1] https://github.com/learnbyexample/scripting_course/blob/mast...

[2] https://mywiki.wooledge.org/BashGuide

[3] https://ryanstutorials.net/linuxtutorial/

[4] https://www.shellcheck.net/

Re:Bash 5.0 released by abryzak on Monday January 07, 2019 @09:42PM (#18852639)
The man pages for bash are fairly comprehensive and I'd definitely recommend referencing them while writing scripts: https://linux.die.net/man/1/bash
Re:Bash 5.0 released by mishac on Monday January 07, 2019 @09:51PM (#18852681)
The full manual is here. https://www.gnu.org/software/bash/manual/bashref.html GNU projects tend have info pages that are a lot more complete and thorough then their man pages, so that'd be a good place to check too.
Re:Bash 5.0 released by chrisseaton on Monday January 07, 2019 @10:21PM (#18852844)
> GNU projects tend have info pages that are a lot more complete and thorough then their man pages

Why is that? Why don’t they build both from a single source?

Re:Bash 5.0 released by int_19h on Tuesday January 08, 2019 @12:50AM (#18853541)
Because they're structured differently. A man page for an app is just that, a page, with some formatting and sections. Texinfo is more like your typical website - lots of small cross-linked pages with a common index. Consequently, a man page is usually an extended take on --help, while info pages are more like product manuals.

This is the theory. The practice is that GNU mandates Texinfo for its projects, and because documentation tends to be the weak spot for all open source projects, manpages end up the most neglected as a result, which can be quite annoying. Especially since pretty much nobody else uses Texinfo - man pages are good enough for most console apps, and those that need more detailed documentation use Docbook, Markdown etc.

Re:Bash 5.0 released by pbhjpbhj on Tuesday January 08, 2019 @05:41AM (#18854645)
They could easily just serialise the info pages under page headings and have that as an additional manpage.
Re:Bash 5.0 released by SllX on Tuesday January 08, 2019 @01:27AM (#18853715)
Historical difference in style and preferences. GNU’s Not UNIX and all that Jazz, so they have different conventions, some of which would not translate to a man page that well. Plus Texinfo is a tool for generating multiple output formats from a single source.

Typically (unless this changed at some point) the info pages are the canonical reference for GNU projects and the man pages are to accommodate Unix hackers.[1]

[1] https://www.gnu.org/prep/standards/standards.html#Man-Pages

Re:Bash 5.0 released by unhammer on Tuesday January 08, 2019 @03:06AM (#18854073)
Re:Bash 5.0 released by jonahx on Monday January 07, 2019 @10:21PM (#18852837)
Great video to give you the right high-level perspective:

https://www.youtube.com/watch?v=olH-9b3VJfs

Re:Bash 5.0 released by make3 on Monday January 07, 2019 @10:07PM (#18852767)
doing the hackerrank bash series helped me quite a bit
Re:Bash 5.0 released by zwischenzug on Tuesday January 08, 2019 @01:14AM (#18853663)
You can’t beat the man page if you’re patient with it. Or you could try my book:

https://leanpub.com/learnbashthehardway

Re:Bash 5.0 released by speg on Monday January 07, 2019 @09:50PM (#18852676)
Pardon my naivety, but what do you normally use?
Re:Bash 5.0 released by giancarlostoro on Monday January 07, 2019 @09:53PM (#18852694)
I usually do programming in Python and other languages. This particular case it makes sense since I'm taking advantage of command line utilities, otherwise I'm usually just making software myself, e.g. web services or daemons.
Re:Bash 5.0 released by h1d on Tuesday January 08, 2019 @01:03AM (#18853611)
You could learn bash to understand existing scripts but you could also learn fish, which can be a more sane looking script than bash, if you're writing your own.