It's all fun and games until someone [XOFF]

Re:It's all fun and games until someone [XOFF] by d0mine on Sunday June 21, 2015 @09:11AM (#9753279)

Just to be clear, the issue is not in Python: the deadlock will happen if you won't consume the pipe whatever language you use to spawn a subprocess -- the buffer is not in your application -- it is the OS pipe buffer.

I imagine your broken code did something like:

  #XXX WRONG, DON'T DO IT
  p = Popen(cmd, stdout=PIPE)
  p.wait() #XXX deadlock is possible if cmd generates enough output
  output = p.stdout.read()

To avoid the deadlock, it is enough to swap the last two statements:

  p = Popen(cmd, stdout=PIPE)
  output = p.stdout.read() 
  p.wait()

that you would normally write as:

  output = check_output(cmd) # 2.7+

If you need to consume more than one pipe separately then you should consume them concurrently and therefore you need threads or async.io.

`Popen.communicate()` can read from both pipes concurrently (as you've mentioned) and it is always available. It was available even in Python 2.4 that introduced the subprocess module.

[ Parent | on HN ]

Re:It's all fun and games until someone [XOFF] by pjungwir on Monday June 22, 2015 @03:04PM (#9760481)

Yep, I've hit this in Perl and Ruby and even once wrote a class to do something like Python's `communicate`. It's especially trappy if you want to run a command and both send stuff to its stdin and read stuff from its stdout. Bash makes it so easy, but it's a fraught with danger anywhere else!

[ Parent | on HN ]