Re:Greplin opensources Lucene Utils and Bloom Filters
by smanek on Wednesday April 13, 2011 @06:14PM
(#2444470)
We originally used mmap too - but it didn't work very well. First, Java has some rather serious mmap limitations, (http://bugs.sun.com/view_bug.do?bug_id=4724038) - and for some reason, we saw occcasional data corruption (which we haven't seen since we moved to the current system).
[ Parent
| on HN ]
Re:Greplin opensources Lucene Utils and Bloom Filters
by jwr on Thursday April 14, 2011 @02:59AM
(#2445930)
Have those mmap limitations really impacted you? And the corruption — I'm assuming that was with read/write mappings that you wrote to?
I'm asking, because we're getting great mileage out of mmap in Clojure, albeit for read-only mappings. And that's in a search engine :-) Using mmap for large data is great, because you avoid enlarging your heap and the garbage collector doesn't even have to care about your data.