Show HN: Turn any website into a knowledge base for LLMs

Re:Show HN: Turn any website into a knowledge base for LLMs by tompec on Monday July 29, 2024 @09:09PM (#41105222)

I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications.

It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically.

I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by blackeyeblitzar on Tuesday July 30, 2024 @02:54AM (#41106660)

Is there a way to deal with websites where you need to login? Like subscription based sites?

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by tompec on Tuesday July 30, 2024 @05:21AM (#41107412)

Unless you own those sites, I'm afraid that's not going to be possible.

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by brianfryer on Thursday August 01, 2024 @12:17AM (#41126112)

But if I do own those sites, there’s a way?

[ Parent | on HN ]