No Space Left on Device

Re:No Space Left on Device by oxplot on Tuesday September 06, 2016 @03:19AM (#12434174)

Standard practice to store a large number of files but in such a way that they don't end up in a single directory, is to create a crypto hash of your object names and use the first n characters as the directory name. Properties of hash functions will ensure that the files are distributed evenly between all the directories.

git is an example of a program that uses this technique (among others) to store blobs. In case of git, the blob names are already SHA-1 hashes.

[ Parent | on HN ]

Re:No Space Left on Device by majewsky on Tuesday September 06, 2016 @04:26AM (#12434398)

You don't actually need a cryptographic hash here [1]. I'd rather use a fast hash like Murmur instead.

[1] Exception: I don't know how prone Murmur is to hash collisions, so if collision-based attacks are part of your threat model, a randomly salted cryptographic hash might be appropriate.

[ Parent | on HN ]