Lucene doesn't have to commit document updates to make them searchable. You just need to grab the IndexReader via the IndexWriter.getReader() method instead of opening an IndexReader on the index directory.
We use this for NRT indexing currently on our site and it works fine - there's a small (500 millisecond or less) delay in changes appearing but it's close enough for us.
We don't use Solr, though - it's a custom search engine based on Lucene.
Its not the only way to do it. Twitter doesn't do it that way, neither does LinkedIn. They have their own Search Engines as well build around Lucene. I haven't looked at them in a while, but I don't think they were commiting every document. They also get their content from their users.
FYI, LinkedIn open-sourced the work they did to build real-time indexing/search on top of Lucene. It's called Zoie (http://javasoze.github.com/zoie/), I've played with it and it's nice from a performance point of view...
Correct, Twitter in fact did a lot of work on the Lucene core specifically for this. The OP was talking about standard Solr though.
Re:Realtime Search: Solr vs Elasticsearch
by true_religion on Tuesday May 31, 2011 @01:35PM
(#2603551)
What's IndexTank using as a backend?
IndexTank :) Meaning, we wrote a search engine from the ground up. We probably should write a blog post to explain why. We did use a couple of things from Lucene:
- The tokenizers, to be compatible with the query syntax - The index format for persistence on disk. It's a great format, no point in reinventing that. Analogous to how Open Office can save in .doc format.
This is the default behavior for IndexTank. When an indexing call returns, whatever you indexed WILL show up in search results. We don't give you the option to commit or not commit, we simply designed IndexTank to be fast enough when indexing in real time. I'd like to see us included in that comparison as well.