News for hackers. Stuff that matters.
Re:Show HN: Open-source Rule-based PDF parser for RAG by StrauXX on Wednesday January 24, 2024 @02:29AM (#39114599)
Last I used it, Azure Document Intelligence wasn't all that smart about choosing split points. This seems to implement better heuristics.
Re:Show HN: Open-source Rule-based PDF parser for RAG by infecto on Wednesday January 24, 2024 @08:04AM (#39116778)
What is a split point? I use Textract a lot and from my testing, always beats out any of the open source tooling to extract information. That could also be highly dependent on the document format.
Re:Show HN: Open-source Rule-based PDF parser for RAG by batch12 on Wednesday January 24, 2024 @08:12AM (#39116853)
I think it is a reference to the place a larger document is split into chunks for calculating embeddings and storage.
Re:Show HN: Open-source Rule-based PDF parser for RAG by asukla on Wednesday January 24, 2024 @09:38AM (#39117807)
I wrote about split points and the need for including section hierarchy in this post: https://ambikasukla.substack.com/p/efficient-rag-with-docume...

All this is automated in the llmsherpa parser https://github.com/nlmatics/llmsherpa which you can use as an API over this library.