> Given that I have images and their types, could I "train" the AI with these million documents to recognize a new image that might come in?

Do I understand this right:

1. You have lots of documents as images, and their type (brief, motion, court order)

2. You get a new document, as an image.

3. You want to assign a type to this new document.

Is that right?

What's your goal in terms of quality? A key way of thinking about this is:

1. What's the risk/cost if you mis-classify a document?

2. What's the risk/cost if you fail to classify a document?

Are the documents typically very structurally different? Could I probably tell them apart without wearing glasses? Or are they largely the same, but with nuanced differences in the text?

That is exactly right.

As far as quality, if I classify the document wrong today (or fail to classify it), it's not that big of a deal to the system as a whole - but users will be "very" annoyed and have to correct it.

> Are the documents typically very structurally different? Could I probably tell them apart without wearing glasses? Or are they largely the same, but with nuanced differences in the text?

The documents are structurally reasonably different, but not "very" different. If a trained human looks at it, they would be able to tell them apart. There are exceptions, of course. For instance, if the attorney doesn't follow accepted convention, but those are rare.

While there are many benefits of end-to-end training, I don't think it might be best suited for this case. This is because we already know that the only useful feature in the document is the text, and not the contours or textures. Theefore wasting neurons in your neural network to learn the wastefulness of this features is just waste of resources. Furthermore, you benefit from even a larger corpus of learned data, which the char recognition engine has been trained on.

Thus, I was looking for a more advanced approach.

OCR is a better understood problem than a general neural net, so I think it's likely easier to improve its quality that to superseded the quality with image-based recognition.

With OCR, you get bits and pieces of information, but because I don't know what the type of the document is, it is difficult to determine where, structurally speaking, this information resides on a page.

If I could use AI to determine the type of the doc, I would know the structure of the document and I could then use OCR to pinpoint specific information on the page.

Most court documents are created from templates.

Leaving the quality part aside -- this job itself is easy to parallelize in that you can split it up by document or by page.

Open option is to run each job in Lambda asynchronously, with the input being a URL to the page or the full document, and have the job call back to you with the text of the page (or put it on S3 as a text file, or add it to a message queue, or whatever works). Regarding splitting: we've been using a python wrapper + pdfium for splitting PDFs into page images on Lambda, with excellent results.

To make the Lambda function, you'll either have to build e.g. Tesseract such that it fits into a 50MB zip, or download it while the Lambda function executes. LambCI has a set of docker containers that they've made for simulating lambda, and the "lambda:build" container makes building things easy and repeatable: https://github.com/lambci/docker-lambda. In a pinch, you can build on an Amazon Linux EC2 instance and it should work on Lambda, but you will have to be more careful about dynamic linking.

As another option: I'm not sure if it's been mentioned, but you can also try a ready-made OCR service before packaging up Tesseract, like this one: https://algorithmia.com/algorithms/ocr/SmartOCR.

So anyway, the performance part has good solutions, at least.

For fixing the accuracy: I know next to nothing about approximate string matching, but perhaps it would then be possible to do a fuzzy search over the text using something similar a Levenshtein automaton: https://en.wikipedia.org/wiki/Levenshtein_automaton.

You may also want to take a look at this: https://en.wikipedia.org/wiki/Bag-of-words_model

More broadly, I'm sure that there are text-based document classification methods that are robust against sloppy OCR. It may just take some research on the main approaches people take to document classification -- it's not my area, but my understanding is that this is typically approached with statistical methods. Otherwise your spam filter would get defeated by typos.

Disclaimer: pretty new to ML

But I know next to nothing about AI and ML - that's why I was asking this question.

Often ML problems way more experimental than "normal" coding. I would first try modelling it as a classification-problem and just do some cross-entropy validation to check the performance of the model. If it's useful, go with it, if not back to the drawing board. You will need some serious computing power, so either buy some GPUs or use the cloud.

You could train a random forest based on the inputs of OCR and NN, if you want to get total ML. You would gain some interpretability (i don't know whether thats important, but i would guess it might)

I am sorry that I can't give you a more concrete answer, these are just ideas. They are probably wrong. Like i said, i am a beginner and also don't really know the problem.

Edit other idea: If you know the location of the relevant OCR-text, you could use the following approach: Use the NN for classification. It will return probability-like values for every category. Take the top 2 (or 3, or every top until they add up to 70 percent...idk). Then do some OCR for every category you have to check. If one is positive you have your result, if not run the others.

I've had success with Clarifai's [0] custom CV model API in the past. You basically upload batches of labeled images to train a model, and then you can submit new images for classification.

Of course, I have no idea how effective it would be for your documents. Obviously it depends on how visually distinct the different types are.

[0] https://www.clarifai.com

From https://developer.clarifai.com/quick-start/

Seems simple to train

  // add inputs with concepts
  app.inputs.create([{
    "url": "https://samples.clarifai.com/dog1.jpeg",
    "concepts": [
      { "id": "cat", "value": false },
      { "id": "dog", "value": true }
    ]
  }, {
    "url": "https://samples.clarifai.com/dog2.jpeg",
    "concepts": [
      { "id": "cat", "value": false },
      { "id": "dog", "value": true }
    ]

  // predict the contents of an image by passing in a url
  app.models.predict(Clarifai.GENERAL_MODEL,  
      'https://samples.clarifai.com/metro-north.jpg').then(
    function(response) {
      console.log(response);
    },
    function(err) {
      console.error(err);
    }
  );

Is that possible with AWS AI?

> Finally, we provide AI engines, a collection of open-source, deep learning frameworks for academics and data scientists who want to build cutting edge, sophisticated intelligent systems, pre-installed configured on a convenient machine image.

Which is to say anything that TensorFlow can do, AWS AI can do. So yes. But is it possible without implementing it yourself? It looks like Amazon Rekognition may be able to do what you're looking for, but I'm not certain. You'll have to research that one.

If not AWS, how could I get started elsewhere? Every tutorial I've looked at deals with a CVS data file, not an image.

Could you point me to a resource that has an example of how to train AI with image documents?

You can make it more performant by downsizing the documents, but obviously this is incredibly GPU-intensive to train. I second the suggestions to just use OCR.

Look at the text API which parses text from an image and gives you it's pixel location.

More than happy to chat with you