Re:Show HN: R2R – Open-source framework for production-grade...
by magundu on Monday February 26, 2024 @09:13PM
(#39519293)
What is content aware chunking?
[ Parent
| on HN ]
Re:Show HN: R2R – Open-source framework for production-grade...
by Art9681 on Monday February 26, 2024 @09:33PM
(#39519449)
Using content appropriate delimiters to create the chunks. If its a Python document, split it at the proper delimiters. If its an HTML document, JSON, etc. We wouldn't want to split a chunk in the middle of a function or paragraph. Or maybe we would. But that is what OP is refering to I believe.
Re:Show HN: R2R – Open-source framework for production-grade...
by chiccomagnus on Tuesday February 27, 2024 @07:07AM
(#39523076)
That's a good point, and documents too needs different chunking techniques. You don't want to split a word file the same way you split an excel...