Re:Ask HN: Best internal tools your startup built?
by reacharavindh on Wednesday November 29, 2017 @03:28AM
(#15805456)
How is it different from using pssh and pass "du -sh /foo/bar" farmed out to all nodes?
Re:Ask HN: Best internal tools your startup built?
by geophile on Wednesday November 29, 2017 @09:46AM
(#15807139)
Because in a 100-node cluster, you get 100 du results. With osh you can post-process cluster output in arbitrary ways, e.g. summing.
Also:
- Piping of Python objects instead of strings.
- Integrated database access (SQL results -> stream of Python tuples).
- Support for arbitrary Python processing.
I needed to operate on all nodes of the cluster. cssh opens a console on each node, and echoes commands to each. But that doesn't scale well past about 12 nodes, doesn't combine results from nodes, it really didn't work for me.
I wrote a tool I called Object Shell, or osh. The idea is to combine commands by passing data over pipe-like connectors. Osh is written in Python, and the commands are customized by Python functions. E.g., to find all the files recursively inside the current directory, and add up all the sizes:
osh: Invokes the toolls -rf: List files (f) recursively (r), yielding a stream of Python file objects.
^: Pipe the stream to the next command.
f: Apply a function to each item in the input stream, writing function output to the output stream.
'file: file.size': A function that returns the size of a file.
red +: Reduction using +, combining all the numbers coming in.
$: Print result to stdout.
And then to do this for all files under /foo/bar, in all nodes of a cluster named mycluster, and to do a cluster-wide sum:
@mycluster [ ... ]: Distribute the bracketed code to each node of mycluster.f 'host, sum: sum': The distributed command returns results that include the host name; drop the host name.
red +: Sum the sums from each node.
Osh is available here: https://github.com/geophile/osh