That first one can be done with teseract ocr pretty cheaply. I think a lot of people just don't know how much software can do. You just need to standardize the form and then there's a human data validation step in there.

Several years ago I came across a pile of interesting-to-me US government data that consisted of scanned PDFs of paper forms (some filled out by hand, some with a typewriter, more recently with a computer). The paper forms change in layout and level of detail a few times over several decades, there are hundreds of sub-schema variations within each time period (yes, that means that a particular variant might have a single record from 1983, a dozen scattered over time, or thousands across several decades), some variations are due to a simple category misspelling, others are more substantial.

Some of the scanned forms have marginal notes (probably okay to ignore these), but I've noticed some cases where the form had prefilled data crossed out with handwritten data replacing it.

Please let me know if a system exists that can deal with this idiosyncratic mess and coaxed to produce structured data of some sort without basically using MTurk behind the scenes. I would be interested in re-releasing the data into the public domain.

Can you not use software for the 80% that are amenable to it and humans for the other 20%?

The challenge we faced (for the two days we tried to hack together a solution) wasn’t so much the OCR but the fact that we couldn’t standardize the form as we were onboarding people with the documents from other banks.

Our requirement was that the the software would need to read the number of separating lines in the table and output accordingly.

Maybe it can do this, idk. I was an intern at the time.

We were using tesseract for our attempt too.

I worked somewhere that did something similar. Their product would scan and extract data from documents. Even working with the banks and companies who produced the documents, there were very stringent requirements (like identifying marks on pages to tell what document type we are looking at, pages being fed in the correct order, etc). Plus the ability to handle degraded, damaged, or bad copies. It's apparently one of those things that seems easy at a glance but the edge cases end up employing a whole department (they had people to review the scans and press OK or correct the data). It was a legacy system and I think a newer rewrite could get further but I'm wary about saying it could be fully automated.

90% automated is sometimes 90% as good as fully automated. It's not always the case, there are cases where hitting that 100% enables you to do things that were just not possible at all before, but sometimes it is. Recognizing those cases where a 90% solution is a 90% improvement at a 10% cost is a valuable skill.

Yes it's also part of the reason I prefer coding for myself. I can ignore a lot of "what ifs" and edge cases because I'm always there to pick up the mess if something happens. I get to focus on the stuff that makes an impact and forget the idiot-proofing :)

Or said another way, it can be fun and also effective to "Do Things That Don't Scale" [http://paulgraham.com/ds.html].

There are commercial OCR software packages that can do it. FineReader is one example (and it's been around for ages).

FineReader seems like it would have done the job.

We used tesseract ocr, plus other image processing libraries to improve ocr's chances, and although we were close to usable results there were still many challenges. One major problem was that different data sources had completely different page layouts, and often times the original documents were out of reach and the client only had available the scans of fax transmissions!

The bottom line was that we were able to achieve very good results consistently, but the margin of error was not acceptable for our specific use case.