News for hackers. Stuff that matters.
Re:Show HN: Turn any website into a knowledge base for LLMs by kaycebasques on Wednesday July 31, 2024 @09:14PM (#41125282)
The more detail, the better. If `<section>` elements are found you chunk those? Do you do it recursively or do you stop after a certain level? And when section elements don't exist, you use `<h1>`, `<h2>`, etc. to infer logical chunks?
Re:Show HN: Turn any website into a knowledge base for LLMs by tompec on Wednesday July 31, 2024 @10:50PM (#41125745)
Having looked at a lot of HTMLs, I noticed that sections are not really the default. I rely on headings (h1, h2, ...) to chunk each pages. Each chunk has its heading hierarchy attached to it. There are a lot of optimizations that could be done at that level.
Re:Show HN: Turn any website into a knowledge base for LLMs by chasd00 on Thursday August 01, 2024 @09:26AM (#41128858)
i'm just guessing but i would think following whatever semantics leads to the highest search rank in google's algorithm would be what you're most likely to find out in the wild.