Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

Re:Show HN: DeepHN – Full-text search of 30M Hacker News pos... by arafalov on Tuesday April 13, 2021 @11:43AM (#26793903)

It seems you tried MMapDirectory and then commented it out. https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneB...

So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches.

Could you share the reasons, MMapDirectory did not work for you?

[ Parent | on HN ]

Re:Show HN: DeepHN – Full-text search of 30M Hacker News pos... by wolfgarbe on Tuesday April 13, 2021 @12:02PM (#26794261)

"NIOFSDirectory and MMapDirectory implementations face file-channel issues in Windows and memory release problems respectively. To overcome such environment peculiarities Lucene provides the FSDirectory.open() method. When invoked, it tries to choose the best implementation depending on the environment." https://www.baeldung.com/lucene-file-search

[ Parent | on HN ]

Re:Show HN: DeepHN – Full-text search of 30M Hacker News pos... by arafalov on Tuesday April 13, 2021 @12:12PM (#26794500)

Right. I found you are running on Windows after posting the comment. Makes sense, though there were some fixes in the latest Lucene I believe.

The Lucene's own benchmarks are at: https://home.apache.org/~mikemccand/lucenebench/ , though I admit to not know enough about benchmarking to form a strong opinion.

Either way, good luck with the project/service. More competition is always great. The open-source components look interesting too.

[ Parent | on HN ]