News for hackers. Stuff that matters.

Ask HN: How to search through 1M scanned TIF images?

Here's my problem: I've got a corpus of over 1 million scanned text documents in tif format. No OCR has been performed.

What is industry best-practice for indexing and searching through these documents?

Does anyone have any first-hand experience with commercial and/or open-source solutions that are performant on this type of problem?

How about experience with ranking algos other than tf-idf?

I'm sure this is a common problem at the enterprise level. For example, just today, the WSJ published an article mentioning that over "100,000 pages of documents [have been collected from] the companies and agencies involved in the" oil spill: http://online.wsj.com/article/SB10001424052748703339304575240210545113710.html?mod=WSJ_hpp_LEFTTopStories

What is the quick, easy, and reliable way to search through these single-serving corpora?

( 6 comments | 11 points | on HN )
Re:Ask HN: How to search through 1M scanned TIF images? by hga on Wednesday May 12, 2010 @04:54PM (#1342237)
I used to do this 15 years ago and as far as I know the approach is still the same:

Run the images through a fixup program that will among other things fix scanning skew.

OCR it, of course. If you can afford it I believe that this is still the best system: http://www.primerecognition.com/augprime/prime_ocr.htm

It can use multiple OCR engines and gets around a nasty bug in the one that has the best quality and speed.

You might want to hire a specialist company for the above tasks. Litigation support companies are one possibility but the lawyers they work for usually have a serious budget.

Then you need to put it into a full text engine that's ideally flexible WRT OCR errors. I.e. the old standard of an inverted index is subpar.

Good luck.

Re:Ask HN: How to search through 1M scanned TIF images? by esers on Wednesday May 12, 2010 @04:53PM (#1342236)
What I'm getting at here is: What is the SugarCRM of Enterprise Search?

I've tried Googling for "DIY enterprise search", but I haven't come up with anything.

All I've found are these very heavy (and very expensive) systems from Autonomy, etc...

Re:Ask HN: How to search through 1M scanned TIF images? by jacquesm on Wednesday May 12, 2010 @04:39PM (#1342191)
You should probably read this first:

http://en.wikipedia.org/wiki/Document_management_system

And then branch out from there.

Re:Ask HN: How to search through 1M scanned TIF images? by WestCoastJustin on Wednesday May 12, 2010 @05:05PM (#1342263)
Re:Ask HN: How to search through 1M scanned TIF images? by trimber on Sunday May 16, 2010 @06:38AM (#1351802)
I believe most enterprise systems are fairly expensive. Maybe you would consider building your own system? I have done a similar thing, by writing a plugin for Google Desktop Search that indexes TIF files. Writing a plugin is pretty straightforward. Now for the OCR part, there is only a few Open Source OCR engines, the most popular being Tesseract. The quality of Tesseract's results is pretty good and I believe is sufficient for many systems.
Re:Ask HN: How to search through 1M scanned TIF images? by keefe on Sunday May 16, 2010 @03:07PM (#1352572)
Raw Tesseract is a pain to use. http://code.google.com/p/ocropus/ This is much easier to deal with and uses Tesseract as the default engine - this is a google funded project btw.