News for hackers. Stuff that matters.
Re:Parallel Locality Sensitive Hashing by herrherr on Friday May 30, 2014 @04:17PM (#7823767)
Actually I'm also using lmdb (together with Python/numpy) :) Added an email address to my profile, would be happy to exchange some experiences.
Re:Parallel Locality Sensitive Hashing by hyc_symas on Saturday May 31, 2014 @02:05AM (#7825391)
100GB isn't that big a deal. If you have at least 16GB of RAM it should be a breeze. There are much larger data sets in OpenLDAP in production around the world.

But I wouldn't choose python for large scale data processing work. The python CPU/memory overhead is like 100:1, compared to C. (This is why I worked on rtorrent and ditched the original bittorrent client ASAP, and why I hate bitbake....)

Re:Parallel Locality Sensitive Hashing by herrherr on Saturday May 31, 2014 @03:35AM (#7825491)
First of all, thanks for open sourcing lmdb :)

The biggest problem currently is actually degrading performance, although I'm almost 100% sure that this isn't caused by lmdb itself, but rather by the bindings I've tried.

In the end, doing it directly in C is probably the only thing that will actually work.