News for hackers. Stuff that matters.

A comment on Show HN: Unriddle – Create your own GPT-4 on top of any document. Parent comment

Re:Show HN: Unriddle – Create your own GPT-4 on top of any d... by wufufufu on Monday April 03, 2023 @01:58PM (#35428457)
Oh so there is pre-processing to find the useful portions? What are you using for the pre-processing?

I feel that it's inevitable that OpenAI et al. will be able to handle large PDF documents eventually. But until then I'm sure there's a lot of value of in this kind of pre-processing/chunking.

[ Parent | on HN ]
Re:Show HN: Unriddle – Create your own GPT-4 on top of any d... by naveedjanmo on Monday April 03, 2023 @02:07PM (#35428599)
Yeah I think you're right - the 32k context window for GPT-4 (not available for everyone yet) is already enough for research papers. I'm using a library called Langchain, there's also LlamaIndex.
[ Parent | on HN ]