yeah sure subprocess.check_output(). If you ever dealt with much perl you know how much more pleasant and easy launching processes was in that language - like shell. Python is great, I'm not on the "bash python" wagon. Launching processes, getting the output, munging it and shoving it at another process is more fiddly and less natural in python. If you want to use that as evidence that I'm a garbage developer go ahead, I realise I'm not writing a carefully researched paper here with examples.

In Perl you literally took the part of your shell pipeline, quoted it and opened it like any other file.

    ./output_generator | wc -l
    becomes

    open(SRC, "output_generator|");
    open(WC, "|wc -l);

And you do whatever you want with those filehandles. It's been ages since I wrote any perl. I miss it.

You might like the python "sh" library [0]. It's not perfect, but it sure does remove the visual noise around subprocess.

[0] https://amoffat.github.io/sh/

> yeah sure subprocess.check_output()

It's deprecated and does something quite different.

> If you ever dealt with much perl you know how much more pleasant and easy launching processes was in that language - like shell.

I'm sure it is, but you're missing my point.

> If you want to use that as evidence that I'm a garbage developer go ahead

Well that escalated quickly.

> open(SRC, "output_generator|");

> open(WC, "|wc -l);

>

> And you do whatever you want with those filehandles.

Again python does roughly the same, just with more overhead: a trailing pipe is an "stdout=PIPE", an input is a "input=<whatever>", and you access / forward stdout explicitly:

    src = Popen('output_generator', stdout=PIPE)
    wc = Popen(['wc', '-l'], input=src.stdout)

And as the sibling notes, for shell replacements you can use the sh library to lower the syntactic overhead of popen.

Doesn't scale well when you have to stitch more than two commands with pipes, which is a very common use case for Unix CLI utilities.

I found this problem as well when doing a direct port including pipes from bash to python. It was the first iteration into python and I was lacking time to do a proper port into python native.

The python code to do piping ends up as longwinded and the plumbing of pipes ends up a massive headache so I wrote tidycmd to overcome that issue https://github.com/laurieodgers/tidycmd

Don't see why it "doesn't scale well", the overhead is roughly constant: use the stdout of one program as the input of the next.

That's not as easy as

    cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' 

You end up writing too much code, it's very verbose. Some times symbols are what you want. In fact the biggest progress in the growth of Math happened when they tossed out doing math with words and bought in symbols.

> That's not as easy as

> cat something | grep "this" | cut -f 1 | sed -e 's/.../.../'

Literally none of this is actually useful if you're already in Python:

    (
        line.split('\t')[0].replace(…, …)
        for line in open('something')
        if 'this' in line
    )

> You end up writing too much code

I can believe that if you're calling to external processes to perform operations which are pretty much trivial in the language.

Of course, if you plan to use one $language alone, you can do anything in that $language.

This question is specific to pipes.

> Of course, if you plan to use one $language alone, you can do anything in that $language.

That's missing what I'm noting though, which is that you don't need pipes anywhere near a shell script if you can simply do more of your work in-language. Case in point being that the entire pipeline you cited as an issue has no reason to exist outside of a shell or shell script.

That's because regexes are an another pain in Python, You have to write a lot of boiler plate exception handling code. Plus the same boiler plate code for file operations.

It doesn't feel like the language was designed for these tasks.

You forgot setting -e and -o pipefail. In case "something" is missing or can't be read for some other reason, your script will just continue happily without warning.

But oh, if you do set -o pipefail the grep will stop the whole pipeline when none of the lines matches "this". So you have to keep fiddling with ${PIPESTATUS[0]}. And none of pipefail or PIPESTATUS are really portable.

Not so simple after all.