Re:What is full-text search (and why you need it now)
by daten on Wednesday December 08, 2010 @11:18AM
(#1983429)
Does it feel like google is moving further away from this with time?
It's common that I get search results that don't contain the keywords I searched for. The results will also contain the keywords but in different parts of the page, so short phrases like hacker news will match pages with "hacker" in the first paragraph and "news" somewhere else on the page. Google used to recognize a hyphen as meaning "only return results where these two words are joined by a space or puncuation", now it ignores the hyphen and you have to enclose the phrase in quotes to get that behavior.
Even more interesting than a simple full-text search would be a regular-expression full-text search of the internet.
In either case, I'm happy to hear IndexTank is trying to fix this.
Re:What is full-text search (and why you need it now)
by JacobAldridge on Wednesday December 08, 2010 @11:49AM
(#1983568)
I vaguely recall that there used to be a feature (and possibly this was with Altavista before I switched to Google) that allowed you to search for words that were close together. So instead of trying ten different permutations of "like jb hifi in London" I would enter the keywords, and various forms "jb hifi type stores in London", "jb hifi stores in central London" etc would be returned. Or am I dreaming?
Re:What is full-text search (and why you need it now)
by sgk284 on Wednesday December 08, 2010 @02:36PM
(#1984284)
It's my understanding that the amount of code is so small that they just have it all in ram (and maybe less accessed data goes to disk) and do a distributed grep.
Re:What is full-text search (and why you need it now)
by elblanco on Wednesday December 08, 2010 @05:19PM
(#1984882)
Right, that's more or less what all the approaches I've seen use. It's not indexing in the sense of full-text indexing for fast retrieval, all of the data is more or less greped over. It's basically just a full-text search, not an index.
Re:What is full-text search (and why you need it now)
by wedesoft on Wednesday December 08, 2010 @12:37PM
(#1983771)
In the past I used HtDig (http://www.htdig.org/) on my Linux desktop. HtDig's latest release is from 2004 and it doesn't support PDF (only Postscript). Also it is not so easy to set up. More recently I tried Strigi but it seems to be very slow. However it has adapters for many file formats including PDF. I don't use MacOS, so I don't know how about the performance of Spotlight.
I think a fast full-text search for the desktop can be very useful. Collecting all those publications doesn't really help when you can't find them when you need them.
Re:What is full-text search (and why you need it now)
by diego on Wednesday December 08, 2010 @12:02PM
(#1983624)
No, we need to make that more clear on the site. It's our own search engine, written to overcome some of the limitations of Lucene. In particular, update of relevance formulas on the fly so you can experiment without reindexing.
Re:What is full-text search (and why you need it now)
by jhandl on Wednesday December 08, 2010 @01:53PM
(#1984086)
Sphinx does not store data, so you have to get it from your database to display results, it doesn't have query auto-suggest, etc.
But most importantly, you don't have to deal with installing, configuring, managing and scaling a sphinx setup. IndexTank is cloud-based, you don't have to deal with servers, RT indexes, hadoop, and so on. We worry about all that.
Re:What is full-text search (and why you need it now)
by sandGorgon on Wednesday December 08, 2010 @12:57PM
(#1983864)
ahh.. i wondered when that would come along.
I had a thought that relevance should be computed like a "stream database", but with an index rather than a data store behind it.
Maybe something like Streambase - but written in Clojure - on top of the lucene index.
Take it one step further, where the entire index is expressed as s-expressions (i'm a bit out of depth here) - you can basically write a mapreduce job on the index.
However, I'm not so sure that computing relevance can be parallelized - i.e can it be broken down as opposed to be computed as a global problem ?
Re:What is full-text search (and why you need it now)
by riffraff on Wednesday December 08, 2010 @01:30PM
(#1983991)
what do you mean by "relevance formula"?
You can implement your own Similarity measurement in lucene and change it at query time, so I'm guessing you mean something else?
Re:What is full-text search (and why you need it now)
by diego on Wednesday December 08, 2010 @01:38PM
(#1984022)
You can define arbitrary mathematical functions based on document variables stored in memory, and that you can change in real time. You can have as many of these as you want.
Re:What is full-text search (and why you need it now)
by riffraff on Wednesday December 08, 2010 @03:49PM
(#1984560)
please pardon me but I still confused: why can't you do this with the standard lucene?
It kind of seems like what solr does with function queries[0]. Is the difference in what the variables are?
Re:What is full-text search (and why you need it now)
by santip on Thursday December 09, 2010 @05:24AM
(#1986889)
The main difference is that you can update the document variables without fragmenting the index. In Lucene you need to reindex the document (i.e. delete it and write a new one). If you update your relevance variables an average of 10 times per document it can get really fragmented with Lucene.
It's common that I get search results that don't contain the keywords I searched for. The results will also contain the keywords but in different parts of the page, so short phrases like hacker news will match pages with "hacker" in the first paragraph and "news" somewhere else on the page. Google used to recognize a hyphen as meaning "only return results where these two words are joined by a space or puncuation", now it ignores the hyphen and you have to enclose the phrase in quotes to get that behavior.
Even more interesting than a simple full-text search would be a regular-expression full-text search of the internet.
In either case, I'm happy to hear IndexTank is trying to fix this.
If you type "cat AROUND(5) dog" into google, it'll return the top results that contain anywhere from 0 to 5 words between cat and dog.
http://www.googleguide.com/wildcard_operator.html
UPDATE: and someone just posted about the AROUND() operator, http://news.ycombinator.com/item?id=1983930
It would seem to me that indexing data in such a way as to support arbitrary regex would be difficult. Anybody know of an existing system?
I think a fast full-text search for the desktop can be very useful. Collecting all those publications doesn't really help when you can't find them when you need them.
But most importantly, you don't have to deal with installing, configuring, managing and scaling a sphinx setup. IndexTank is cloud-based, you don't have to deal with servers, RT indexes, hadoop, and so on. We worry about all that.
I had a thought that relevance should be computed like a "stream database", but with an index rather than a data store behind it. Maybe something like Streambase - but written in Clojure - on top of the lucene index.
Take it one step further, where the entire index is expressed as s-expressions (i'm a bit out of depth here) - you can basically write a mapreduce job on the index.
However, I'm not so sure that computing relevance can be parallelized - i.e can it be broken down as opposed to be computed as a global problem ?
http://indextank.com/documentation/function-definition
(Feedback on this documentation is more than welcome, it's a work in progress!)
http://wiki.apache.org/solr/FunctionQuery