Ive found it harder than expected to simply port over a bash script to python. Mostly because the syntax for 'run command c with arguments x y is much worse than bash. And that is before you want to handle output from the command.

It would be super nice if there was a package offering "native" shell experience in python. Say

  from pyshell import sh, ShellError  
  
  def func(a,b)  
      output, retcode = sh("df -k | awk '{print $5}'")  
      if retcode == 0:  
          return output  
      else:  
          raise ShellError(output,retcode)

The sh module is really excellent, and it lets you take the best parts of bash (the external commands, pipes) and ditch the bad parts (string handling, general programming). I think it’s what you’re looking for.

https://amoffat.github.io/sh/

I wrote https://github.com/tarruda/python-ush for my own scripts. It was inspired by "sh", but my version supports native "lazy" pipelines (using python operator overloading) and other nice features not provided by sh. Also it works seamless between Unix/windows

I was also planning to extend it to support python3 asyncio, but never got around it

https://shell.readthedocs.io/en/latest/ looks promising, will try next time before writing "just a small script" in bash (small scripts always grow and you pile on dirt on them when starting in bash).

This looks great, thanks! It looks very easy compared to subprocess.run and friends. I hope it'll help convincing my colleagues to default to python instead of bash (or, heavens forbid, csh)

Check out https://github.com/tarruda/python-ush

It provides an idiomatic shell scripting experience for python.

For example, it uses the "pipe" operator overload to allow complex pipelines to be created (pipes are created using platform native pipes, so there's no performance difference from a shell pipeline). The same operator can also be used for redirections

Isn't this what os.system(...) does?

By the way, why not put the raise inside sh?

EDIT: Ok, the difference is that os.system does not return stdout. But you can use the subprocess package to capture stdout (and stderr as well). If you want pipe syntax, you can invoke e.g. bash -c "...", where the dots are your command. But the problem with this is that it's prone to injection attacks, unless you are very careful. I guess the same warning applies in your original "sh" example.

Fabric'll do that for you (or rather Invoke, I guess I should say as part of the Fabric 2 changes)

    >>> from invoke import run
    >>> date = run("date")
    Sat Apr 13 11:09:18 PDT 2019
    >>> dir(date)
    ['__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'command', 'encoding', 'env', 'exited', 'failed', 'hide', 'ok', 'pty', 'return_code', 'shell', 'stderr', 'stdout']
    >>> date.return_code
    0
    >>> date.stdout
    'Sat Apr 13 11:09:18 PDT 2019\n'
    >>> do_they_exist = run("grep banana /etc/passwd")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/pgraydon/Envs/python3/lib/python3.7/site-packages/invoke/__init__.py", line 46, in run
        return Context().run(command, **kwargs)
      File "/Users/pgraydon/Envs/python3/lib/python3.7/site-packages/invoke/context.py", line 94, in run
        return self._run(runner, command, **kwargs)
      File "/Users/pgraydon/Envs/python3/lib/python3.7/site-packages/invoke/context.py", line 101, in _run
        return runner.run(command, **kwargs)
      File "/Users/pgraydon/Envs/python3/lib/python3.7/site-packages/invoke/runners.py", line 271, in run
        return self._run_body(command, **kwargs)
      File "/Users/pgraydon/Envs/python3/lib/python3.7/site-packages/invoke/runners.py", line 404, in _run_body
        raise UnexpectedExit(result)
    invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!
    
    Command: 'grep banana /etc/passwd'
    
    Exit code: 1
    
    Stdout: already printed
    
    Stderr: already printed

If it gets a non-zero code back (like grep does when it fails to get a match), it raises an UnexpectedExit exception.

This is a simple example, but usually you have command line arguments, the output can be large, and you might have input to process. As soon as you have that, you arrive at a more complex API, but one where you don't shoot yourself in the foot so easily (e.g. with special characters in file names and friends).

I disagree. If I’m porting a shell script to another language then I would want to use as much native code as possible rather than shelling out. Sure, it takes a little more effort, but the output is more idiomatic code that runs faster (fork() is slow).

Fwiw ruby borrows heavily from perl, and can be coaxed into serving as a driver for other commands. See eg:

http://ibmsystemsmag.com/aix/tipstechniques/systemsmanagemen...

https://redpanthers.co/different-ways-to-run-shell-commands-...

Perl/awk like one-liner/filters in ruby:

http://reference.jumpingmonkey.org/programming_languages/rub...

https://nithinbekal.com/posts/ruby-sed-awk/

Of note is that ruby allows "backtick execution":

filename = 'some_file.txt' wordcount = `wc --words #{filename}`

I find that much of the strength of bash/shell comes from the surrounding utilities though. So simply jumping to ruby won't magically let your "shell script" work on both windows and Linux, just because you have ruby installed;there likely not be a similar set of building blocks like "wc" etc. You could of course replicate most via the ruby std lib - and it's arguably easier to write test driven ruby than bash.

Perl is much closer to bash in this regard, mostly use

  qx(some command some parameters some variables)

nice one. this is my favorite way of learning languages. find it really hard going step by step through a book from scratch and my fingers get really itchy and want to skip ahead into solving real problems. my result is usually a lot worse than what you built there.

seems a lot of effort and iterations went into this?

I have seen quite a bit of legacy plumbing like this in production / automation until better (or at least standardized) tooling came along with the agile/devops movement. A lot of this plumbing survived surprisingly long (which is also scary). Often code like this matures over long periods into stable solutions that could even deal with errors well. Makes them hard to replace.

I agree it doesn't belong in production but it looks pretty neat for small internal stuff even in 2019.

It was a lot of work but it was my first real attempt at programming and the topic itself was motivating enough to bring it to completion.

Learning from books doesn't work, you need to have a project with a goal to apply it in.

The horrible honest truth is that the world actually doesn't run on bash, python, C or Java.

It runs on Excel macro's.

This. Some of my Structural engineering professors have invested so much time in Excel they practically have an instance of their mind in it. Really fancy systems too that could do just about anything they needed. Always impressed me.

Your project is a great read!