News for hackers. Stuff that matters.
Re:Hashpipe – Pipe iff the hash matches by kazinator on Friday April 03, 2015 @08:43PM (#9319418)
It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output.

A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram:

   whatever | hashexec <hash> arbitrary --command with args
This "arbitrary" command's standard input is redirected from a temporary file created by hashexec (so no cat-like loop has to execute to feed the data).

Also, since this is for scripts, there could be an argument which limits the size. This could have default value, say one megabyte. Anyone pulling down scripts which are anywhere near one megabyte has to add explicit overrides for the size:

   whatever | hashpipe --max=4M <hash> | ...
The real problem isn't that hashpipe buffers everything, because the scripting language running the script will also do that; rather that because it buffers everything, it can be DDoS'ed with an infinite stream or whatever.

By incorporating a size limit, hashpipe could provide an additional protection measure to the next pipeline element: it protects against content which doesn't match the hash, and against content which is too large.

Idea: the size of the input could be encoded as a few digits of the hash. Then the argument is unnecessary. The hash itself tells you that the script is exactly 6059 bytes long; if you read 6060 bytes, the input is not the right one. Or the program could just stop reading at 6059 and check the hash at that point, and either pass on the 6059 bytes or error out.

Re:Hashpipe – Pipe iff the hash matches by _prometheus on Saturday April 04, 2015 @12:02AM (#9319850)
All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues ? over time we can implement some
Re:Hashpipe – Pipe iff the hash matches by kazinator on Saturday April 04, 2015 @12:14AM (#9319869)
I don't use github; I have my own server with git and cgit.