Re:Web search 30M rows sub-second, cheaply
by lukaslalinsky on Monday November 23, 2015 @04:48AM
(#10613678)
Most comments here are about using an existing search solution, but sometimes inventing your own solution has a lot of benefits.
I needed something similar when I was building https://acoustid.org/ and for the first iteration I used PostgreSQL's GIN indexes, which are nice and easy to work with, but once the index grew beyond some point, it was getting very slow. I spent some time designing a minimalistic inverted index that does just what I need, nothing else, while compacting the data as much as possible, to keep it in RAM, and the results were just excellent. It was super fast compared to the GIN indxes. Now I have 10x more data and it's still as fast as it was before. If you are interested, here is the code - https://bitbucket.org/acoustid/acoustid-index
I needed something similar when I was building https://acoustid.org/ and for the first iteration I used PostgreSQL's GIN indexes, which are nice and easy to work with, but once the index grew beyond some point, it was getting very slow. I spent some time designing a minimalistic inverted index that does just what I need, nothing else, while compacting the data as much as possible, to keep it in RAM, and the results were just excellent. It was super fast compared to the GIN indxes. Now I have 10x more data and it's still as fast as it was before. If you are interested, here is the code - https://bitbucket.org/acoustid/acoustid-index