News for hackers. Stuff that matters.
Re:Postgres full-text search is good enough by messel on Monday December 08, 2014 @04:47AM (#8715687)
Hi Justin, do you have a link to a technical paper on the anonymous? click algorithm. Was curious if you applied it to domains other than products/shopping.
Re:Postgres full-text search is good enough by justin_vanw on Monday December 08, 2014 @05:31AM (#8715799)
I linked to the patent application (which has been denied).

The algorithm is actually very very simple.

1. produce a set of 'features' for every document. This is what you will index into the inverted index. Features can be words from the document, or they can be things like 'price_between_0_100'.

2. Build a set of pairs (search_phrase, clicked_on_document_id).

3. For each search_phrase in your set of pairs, run the search_phrase through your search engine. The set of all results we will call SET_A.

4. SET_B is the set of documents for a search_phrase that have been clicked on. SET_B is a subset of SET_A. Using mutual information, it is possible to determine which features best differentiate documents from SET_B from (SET_A - SET_B) for a specific search_phrase. These are the features you will 'boost' when you see that search_phrase.

As an example, one of the features this pulled out in practice was the word 'Oceanic' for the search_phrase "lost". It basically zeroed in on the fact that the word 'Oceanic' appearing in the description of a product differentiated 'good' results from 'bad' results for the query 'lost'.

Edit: Here is a hard to follow presentation on it: https://docs.google.com/presentation/d/1xzQbQ1cM3GnaS_LoCvmJ...

Edit2: Another presentation, easy to follow but without most of the technical depth: https://drive.google.com/file/d/0B_eNCTyNa0DHcy1UOGpSUk41NDA...