Re:Show HN: Waymark – A workflow engine that compiles jobs i...
by msteffen on Saturday November 07, 2026 @06:33PM
(#50409061)
I guess I could add:
With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers, and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^1]). Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".
We never tried to optimize the single-machine case, for better or worse.
[^1]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)
With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers, and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^1]). Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".
We never tried to optimize the single-machine case, for better or worse.
[^1]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)