Re:Show HN: Turn any website into a knowledge base for LLMs
by tompec on Wednesday July 31, 2024 @08:19PM
(#41124959)
I chunk pages and generate embeddings for each chunk. So there's no real size limit per page.
Re:Show HN: Turn any website into a knowledge base for LLMs
by kaycebasques on Wednesday July 31, 2024 @09:14PM
(#41125282)
The more detail, the better. If `<section>` elements are found you chunk those? Do you do it recursively or do you stop after a certain level? And when section elements don't exist, you use `<h1>`, `<h2>`, etc. to infer logical chunks?
Re:Show HN: Turn any website into a knowledge base for LLMs
by tompec on Wednesday July 31, 2024 @10:50PM
(#41125745)
Having looked at a lot of HTMLs, I noticed that sections are not really the default. I rely on headings (h1, h2, ...) to chunk each pages. Each chunk has its heading hierarchy attached to it. There are a lot of optimizations that could be done at that level.
A key question that the docs should answer (and perhaps the "How it works" page too): chunking. You generate an embedding for the entire page? Or do you generate embeddings for sections? And what's the size limit per page? Some of our docs pages have thousands of words per page. I'm doubtful you can ingest all that, let alone whether the embedding would be that useful in practice.