News for hackers. Stuff that matters.
Re:How Discord Indexes Billions of Messages Using Elasticsearch by tannhaeuser on Friday March 17, 2017 @05:31AM (#13892663)
On the other side of the spectrum, I recently did a mini review of available F/OSS options for full-text indexers /search tools. Requirements: natively compiled, one-shot/single-process per search, no long-running containers so EL, SOLR, and anything else based on mighty Lucene and Java was out of scope.

The reviewed tools were: clucene, Apache Lucy, Indri/Lemur, Xapian, and Zettair. Of those, only Xapian and Zettair could be made to return useful search results within time constraints. Sadly, I didn't get Apache Lucy to build, which along with Xapian is the only option still being maintained. clucene could be built, but delivered erratic results (it found "impeachment" but not "indictment" in the US constitution text), and was too difficult to extend into a full stand-alone app (though it apparently is being used in Informix still).

Re:How Discord Indexes Billions of Messages Using Elasticsearch by rbutcher on Friday March 17, 2017 @09:56AM (#13893984)
Have you seen Bleve? https://github.com/blevesearch/bleve

I have only tried it for an hour or two, and can't speak for its search result quality, but it seems promising.

On my laptop, with an index of 10,000 documents each containing 200 random words, it takes about 110ms to perform a fuzzy search. Half of that is the actual search, the other half is the process/initialization overhead.

Re:How Discord Indexes Billions of Messages Using Elasticsearch by tannhaeuser on Friday March 17, 2017 @11:41AM (#13894802)
Thanks. Will try.
Re:How Discord Indexes Billions of Messages Using Elasticsearch by markpapadakis on Friday March 17, 2017 @06:31AM (#13892894)
FWIW we will likely open source our search engine/IR technology sometime next month (hold up is the documentation and cleaning up some parts of the codebase).

It's written in C++, and the core features are performance, simplicity(operation and extension of functionality) and a sane, simple API (the codebase is quite small as well, at about 14k lines of code).

Its similar in some regards to Lucene, but does away with most of the exotic features and instead is optimised for rich queries support, fast execution and extensibility.

It should be on GH sometime in April.

Re:How Discord Indexes Billions of Messages Using Elasticsearch by tannhaeuser on Friday March 17, 2017 @06:40AM (#13892929)
Looking forward to it. Has it a project name yet?
Re:How Discord Indexes Billions of Messages Using Elasticsearch by markpapadakis on Friday March 17, 2017 @06:44AM (#13892936)
Yes, its called Trinity.
Re:How Discord Indexes Billions of Messages Using Elasticsearch by frik on Saturday March 18, 2017 @02:32AM (#13900197)
Sphinx search is C++, and uses little memory compared to Lucene. It has many advanced features like Lucene.

https://en.wikipedia.org/wiki/Sphinx_(search_engine)

Re:How Discord Indexes Billions of Messages Using Elasticsearch by tannhaeuser on Saturday March 18, 2017 @03:47AM (#13900366)
I didn't consider Sphinx because it's using a RDBMS as store. For the same reason, I didn't check out sqlite-fts etc.

Maybe I should reconsider?

Re:How Discord Indexes Billions of Messages Using Elasticsearch by frik on Saturday March 18, 2017 @07:32AM (#13900831)
Sphinx does NOT use an RDBMS as store. It just provide a convienient interface so you can query Sphinx like you would do a SQL database, just one of several APIs to interact with Sphinx. The data store is very similar to Lucene, a folder with several files.