Re:Show HN: Turn any website into a knowledge base for LLMs
by tompec on Wednesday July 31, 2024 @08:19PM
(#41124959)
I chunk pages and generate embeddings for each chunk. So there's no real size limit per page.
[ Parent
| on HN ]
Re:Show HN: Turn any website into a knowledge base for LLMs
by kaycebasques on Wednesday July 31, 2024 @09:14PM
(#41125282)
The more detail, the better. If `<section>` elements are found you chunk those? Do you do it recursively or do you stop after a certain level? And when section elements don't exist, you use `<h1>`, `<h2>`, etc. to infer logical chunks?
Re:Show HN: Turn any website into a knowledge base for LLMs
by tompec on Wednesday July 31, 2024 @10:50PM
(#41125745)
Having looked at a lot of HTMLs, I noticed that sections are not really the default. I rely on headings (h1, h2, ...) to chunk each pages. Each chunk has its heading hierarchy attached to it. There are a lot of optimizations that could be done at that level.