Ask HN: How to search through 1M scanned TIF images?
Here's my problem: I've got a corpus of over 1 million scanned text documents in tif format. No OCR has been performed.
What is industry best-practice for indexing and searching through these documents?
Does anyone have any first-hand experience with commercial and/or open-source solutions that are performant on this type of problem?
How about experience with ranking algos other than tf-idf?
I'm sure this is a common problem at the enterprise level. For example, just today, the WSJ published an article mentioning that over "100,000 pages of documents [have been collected from] the companies and agencies involved in the" oil spill: http://online.wsj.com/article/SB10001424052748703339304575240210545113710.html?mod=WSJ_hpp_LEFTTopStories
What is the quick, easy, and reliable way to search through these single-serving corpora?
(
6 comments
| 11 points
| on HN
)
Run the images through a fixup program that will among other things fix scanning skew.
OCR it, of course. If you can afford it I believe that this is still the best system: http://www.primerecognition.com/augprime/prime_ocr.htm
It can use multiple OCR engines and gets around a nasty bug in the one that has the best quality and speed.
You might want to hire a specialist company for the above tasks. Litigation support companies are one possibility but the lawyers they work for usually have a serious budget.
Then you need to put it into a full text engine that's ideally flexible WRT OCR errors. I.e. the old standard of an inverted index is subpar.
Good luck.
I've tried Googling for "DIY enterprise search", but I haven't come up with anything.
All I've found are these very heavy (and very expensive) systems from Autonomy, etc...
http://en.wikipedia.org/wiki/Document_management_system
And then branch out from there.
http://open.blogs.nytimes.com/2007/11/01/self-service-prorat...