News for hackers. Stuff that matters.
Re:Evolution of Search Engines Architecture – Algolia Search... by ramoz on Sunday August 08, 2021 @02:44PM (#28109058)
I've scaled large transformer based models that supplement a lucene-based search engine. The architecture supports an ensemble approach where Lucene results are first-class and then we tailor similarity rankings with the models.

It looks a lot like this: https://huggingface.co/blog/bert-cpu-scaling-part-1

We have to store large "index" embeddings on SSDs and use leveldb for value retrievals of the lucene results.

There are no comments on this comment yet.