We've talked to a lot of different developers about these problems and haven't found a general consensus on what features are needed, so we are still evaluating advanced approaches.
For now our implementation is more general and designed to work across a variety of documents. R2R was designed to be very easy to override with your own custom parsing logic for these reasons.
Lastly, we have been focusing a lot of our effort on knowledge graphs since they provide an alternative way to enhance RAG systems. We are training our own model for triples extraction that will combine with the automatic knowledge graph construction in R2R. We are planning to release this in the coming weeks and are currently looking for beta testers [we made a signup form, here - https://forms.gle/g9x3pLpqx2kCPmcg6 for anyone interested]
You gave the example of unstructured PDF, but there are challenges with structured docs as well. We’ve run into docs that are hard to chunk because of this deeply nested and repeated structure. For example, there might be a long experimental protocol with multiple steps; at the end of each step, there’s a table “Debugging” for troubleshooting anything that might have gone wrong in that step. The debugging table is a natural chunk, except that once chunked there are a dozen such tables that are semantically similar when decoupled from their original context and position in the tree structure of the document.
This is one example, but there are many other cases where key context for a chunk is nearby in a structured sense, but far away in the flattened document, and therefore completely lost when chunking.
I'm just curious because we have theorized and seen in practice that extraction is a way to answer questions which require connected information across disparate chunks, like you can see in the simple cookbook here [https://r2r-docs.sciphi.ai/cookbooks/knowledge-graph].
Or do you think this is something that can just be solved with more advanced multimodal ingestion?
It feels like we are inching closer to automating this type of thing, or at the very least brute-forcing it in like the LLM race is trying to do with bigger models and larger contexts.
Will have to play with this over a weekend and see what it might help me with :)
Try it with complex layout documents -> https://pg.llmwhisperer.unstract.com/
If anyone wants to solve for RAG right from loading from source, extraction, and sending processed data to destination/API, try Unstract [2] (it is open-source)
I think the serious problem is most of these LLMs are already built on-top of garbage so you're already the GI and just trying to match that as best you can.
We have a customer who has hundreds of thousands of unstructured and diverse PDFs (containing tables, forms, checkmarks, etc.), and they need to accurately convert these PDFs into markdown to then prepare that data for RAG usage.
Traditional OCR approaches fall short in many of these cases, so we've started using a combined multimodal LLM + OCR approach that has led to promising accuracy and consistency at scale. The RAG system itself is not a big pain point for them, but the accurate and efficient extraction and structuring of the data is.