Re:Lucene's FuzzyQuery is 100 times faster in 4.0
by timr on Wednesday April 27, 2011 @04:04PM
(#2490878)
I don't want to sound like a troll, but Lucene's built-in spellchecker is a toy. If you're doing much more than trying to just plonk in a search engine on your website, you'll probably find it insufficient.
As far as stemming goes, AFAIK, Lucene didn't invent anything. It's a lot like crypto, in that nearly everyone uses public implementations of of stemmers for different languages: http://snowball.tartarus.org/
The previous two points both apply to stop words (you'll almost certainly want to customize your own list, and there are good, curated lists out there for the most common languages), and Lucene's tokenizer options aren't really that smart -- if you want to get into chunking, phrase identification, etc., you're going to have to write it yourself. But then you're going to be working within the Lucene world, tied to their index format, mucking around in the depths of unfamiliar code, etc. That's a big trade-off.
Basically, Lucene is great if you're working on a website and you need a drop-in solution for search, but if your core user experience is search, it has a lot of deficiencies.
As far as stemming goes, AFAIK, Lucene didn't invent anything. It's a lot like crypto, in that nearly everyone uses public implementations of of stemmers for different languages: http://snowball.tartarus.org/
The previous two points both apply to stop words (you'll almost certainly want to customize your own list, and there are good, curated lists out there for the most common languages), and Lucene's tokenizer options aren't really that smart -- if you want to get into chunking, phrase identification, etc., you're going to have to write it yourself. But then you're going to be working within the Lucene world, tied to their index format, mucking around in the depths of unfamiliar code, etc. That's a big trade-off.
Basically, Lucene is great if you're working on a website and you need a drop-in solution for search, but if your core user experience is search, it has a lot of deficiencies.