Show HN: Documind – Open-source AI tool to turn documents into structured data

Re:Show HN: Documind – Open-source AI tool to turn documents... by groby_b on Monday November 18, 2024 @03:18PM (#42176460)

That's not what [1] says, though? Quoth: "As of March 1, 2023, data sent to the OpenAI API will not be used to train or improve OpenAI models (unless you explicitly opt-in to share data with us, such as by providing feedback in the Playground). "

"Traditional methods (PDF parsers with OCR support) are cheaper, more reliable"

Not sure on the reliability - the ones I'm using all fail at structured data. You want a table extracted from a PDF, LLMs are your friend. (Recommendations welcome)

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by niklasd on Monday November 18, 2024 @03:46PM (#42176810)

We found that for extracting tables, OpenAIs LLMs aren't great. What is working well for us is Docling (https://github.com/DS4SD/docling/)

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by soci on Monday November 18, 2024 @06:16PM (#42178239)

agreed, extracting tables in pdfs using any of the available openAI models has been a waste of prompting time here too.

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by emmanueloga_ on Tuesday November 19, 2024 @12:12AM (#42180258)

Haven't seen Docling before, it looks great! Thanks for sharing.

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by emmanueloga_ on Monday November 18, 2024 @07:59PM (#42179086)

> That's not what [1] says, though?

Documind is using https://api.openai.com/v1/chat/completions, check the docs at the end of the long API table [1]:

> * Chat Completions:

> Image inputs via the gpt-4o, gpt-4o-mini, chatgpt-4o-latest, or gpt-4-turbo models (or previously gpt-4-vision-preview) are not eligible for zero retention."

1: https://platform.openai.com/docs/models#how-we-use-your-data

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by groby_b on Tuesday November 19, 2024 @05:01PM (#42188577)

Thanks for pointing there!

It's still not used for training, though, and the retention period is 30 days. It's... a livable compromise for some(many) use cases.

I kind of get the abuse policy reason for image inputs. It makes sense for multi-turn conversations to require a 1h audio retention, too. I'm just incredibly puzzled why schemas for structured outputs aren't eligible for zero-retention.

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by emmanueloga_ on Wednesday November 20, 2024 @12:05AM (#42190973)

Gotcha, from what I could find online I think you are right. I was conflating data not under zero-retention-policy with data-for-training.

[ Parent | on HN ]

Re:Show HN: Documind – Open-source AI tool to turn documents... by pconstantine on Saturday November 23, 2024 @10:54PM (#42225831)

It takes >50 seconds to generate these schemas for some pretty simple use-cases with large enums, for example. Imagine that latency added to each request...

[ Parent | on HN ]