Re:Show HN: Open-source search engine with 2bn-page index
by 6502nerdface on Tuesday September 13, 2016 @12:44PM
(#12489991)
Wow, the contrast between what this engine returns for that query and what google returns is amazing. Literally zero relevant links from the former and only relevant links from the latter. Search relevance is a serious high-science research problem, and it's going to be tough to compete with established players that have probably man-centuries' worth of proprietary research IP and some of the world's best scientists.
Re:Show HN: Open-source search engine with 2bn-page index
by ganeshkrishnan on Tuesday September 13, 2016 @08:05PM
(#12493275)
>Search relevance is a serious high-science research problem
We are working on this scenario at http://www.shoten.xyz using document clustering and apache spark graphx+ giraph.
Ain't been an easy task so far but we have made some headway
A search for "java twitter bot" places more emphasis on "bot" then on Java and then on twitter which is what a tf-idf would do.
A good start like you said but it's miles away even from yahoo or bing.