I’m doing something similar at work. Thousands of contracts, need to extract billing terms/logic. It’s scattered throughout each contract but has no summary, each contract can be hundreds of pages of non applicable legalese. The problem is, once I run it, can I trust the output without manual verification? How will I know if it skipped/missed something from a contract?

I would build a review dashboard that shows you the extracted data on one side and the visual snippet of the contract on the other. Quick to double check, and you still save time over opening the contract and searching for the right section and extracting it by hand or via OCR.

We have done something similar where it highlights the terms, basically each extracted term is linked to a specific highlight in the contract, and has them side by side like you said and you can jump around the contract by clicking on the extracted term. This helps when the agent needs HITL clarification on a term, as it gets you there quick and you can tell it more information to aid interpretation. But, if it missing something completely, I’d never know without reading the whole contract and validating every term I found the agent noticed. We’ve done some training sets and it does help and get better but the large variation of contracts means there’s little standardization.

I think we’re on path to just do random full review audits of maybe 10% to get a baseline confidence level and then just note it as a risk. Still a large manual task, we’ll probably outsource it.