Re:AI for AWS Documentation
by tomhamer on Thursday July 06, 2023 @10:26PM
(#36626270)
Marqo provides automatic, configurable chunking (for example with overlap) and can allow you to bring your own model or choose from a wide range of opensource models. I think e5-large would be a good one to try. https://github.com/marqo-ai/marqo
[ Parent
| on HN ]
Re:AI for AWS Documentation
by d4rkp4ttern on Friday July 07, 2023 @08:08AM
(#36630464)
Using Qdrant doesn’t require docker, like Marqo does (from the README). Any trade offs between the two? Doc chunking is an independent functionality and there are already line that help chunk with overlap etc,and also it’s not hard to roll your own.