Safe ways to do things in bash

Re:Safe ways to do things in bash by freedomben on Monday May 14, 2018 @11:45PM (#17071496)

I personally consider awk, sed, grep, cut, etc. to be "part" of bash. I don't think I ever write a script without invoking those venerable tools at least once. I often have many pipes, such as

  some_command \
    | grep -E 'some.*regex$' \
    | sed -e 's/erase_text//g' \
    | sed -e 's/erase_more//g' \
    | awk '{ print $2 }' \
    | cut -d ':' -f 1

I used to reach for Perl one-liners a lot, and still do sometimes when I need a gross regex that `sed -E` can't handle (See Perl Pie[1]), but Ruby has been making an appearance more and more often for handy one-liners. Roblog has a great post on using Ruby [2].

If the script is complex enough and I'm willing to take a dependency on Ruby, that is usually where I turn. Where that line is does change from time to time. Ruby makes it super trivial to call shell commands and process them easily as well, which makes it easy to integrate into existing scripting [3].

Hopefully that answered your question :-)

[1] http://technosophos.com/2009/05/21/perl-pie-if-you-only-lear...

[2] https://robm.me.uk/ruby/2013/11/20/ruby-enp.html

[3] https://stackoverflow.com/a/2400/2062384

[ Parent | on HN ]

Re:Safe ways to do things in bash by laumars on Tuesday May 15, 2018 @04:08AM (#17072386)

You might already be aware of this, but you can chain multiple sed functions together into one command. eg

    sed -e 's/erase_text//g; s/erase_more//g'

But I appreciate your example there is more for illustrative purposes rather than an actual pipeline you have in production.

[ Parent | on HN ]

Re:Safe ways to do things in bash by zippitydoodah68 on Tuesday May 15, 2018 @04:03PM (#17077359)

These types of pipelines are why bash and the traditional tools are so maligned. Expert knowledge of sed|(g)awk|bash tools is necessary otherwise you end up with companies telling you that you can't use them.

some_command | gawk ' /some.*regex$/ {gsub(erase_text,"");gsub(erase_more,"");split($2,a,":"); print (length(a[1]) ? a[1] : "STRING ERROR")}'

[ Parent | on HN ]

Re:Safe ways to do things in bash by freedomben on Tuesday May 15, 2018 @05:48PM (#17078164)

I'll definitely agree that it can get out of hand. If it's a script/tool that will have many eyes on it and people that need to understand it, bash tools probably aren't the best way to go. I usually turn to ruby in those cases. But even for one off commands, I find myself using pipes like that all the time and I'm the only one that will ever see it, so if it's greek it's no problem :-)

[ Parent | on HN ]

Re:Safe ways to do things in bash by rdtsc on Tuesday May 15, 2018 @01:52AM (#17071928)

> sed -e 's/erase_text//g' \

I like the pattern. Copied it to my notes to try next time I am grepping through logs.

I usually just string a few 'grep -v ignore_this' with pipes but then it ignores the whole line. But I can see how erasing some parts of the line would be very helpful sometimes.

[ Parent | on HN ]

Re:Safe ways to do things in bash by sa46 on Wednesday May 16, 2018 @03:22AM (#17080601)

Drive by code review, you can end a line with the pipe (also && and ||) to avoid a backslash.

    some_command |
      grep -E 'some.*regex$' |
      sed -e 's/erase_text//g' |
      sed -e 's/erase_more//g' |
      awk '{ print $2 }' |
      cut -d ':' -f 1

[ Parent | on HN ]