It makes sense if you have multiple machines to actually distribute the load (files could be on a NFS).

You can send the files through the pubsub directly and just convert a stream of bytes. It is a beautiful pattern, but needs some tools to make it really easy to use (e.g. how to I metadata and bytestream to a pipe)

Sending the actual files would probably be really slow on the free server, due to bandwidth limiting.

As for metadata, you might consider multipart/form-data, or just packing it into the query string (obviously depends on how much data you're talking about). Of course you'd probably need to write that in something other than bash.

EDIT: It's pretty easy to send form-data with curl: https://ec.haxx.se/http-multipart.html

Yes but it is the receiving that mess you up, if you want to use standard shell tools and just stream it. Everything can be solved by using other tools.

# preserve filename for i in a b; do gzip -9 $i; done

# does not preserve filenames while true; done curl $patchbay| mangle | gzip -9 | curl -d@- $patchbay; done

You might be better served with gnu parallel -

  #create the directory structure first
  find . -type f -name '*.flac' |ionice -c3 parallel -j1 'mkdir -p $transcodes{//}' &&
  #now the encodes
  find . -type f -name '*.flac' |ionice -c3 parallel -j4 "ffmpeg -n -v 0 -i {} -c:a libopus -b:a 128k $transcodes{.}.opus"

Note the {.} which would actually only grab the basename.

The ' should be " in first example. Otherwise $transcodes is not expanded.

Actually no, I'm handing that line to parallel which in turn will do the expansion when it invokes.

ls -1 *.mp3 | awk -F. '{OFS = "."; $NF=""; print $0}' | xargs -P4 -n1 -i ffmpeg "{}mp3" "{}ogg"