News for hackers. Stuff that matters.
Re:AI for AWS Documentation by tomhamer on Thursday July 06, 2023 @10:26PM (#36626270)
Marqo provides automatic, configurable chunking (for example with overlap) and can allow you to bring your own model or choose from a wide range of opensource models. I think e5-large would be a good one to try. https://github.com/marqo-ai/marqo
Re:AI for AWS Documentation by d4rkp4ttern on Friday July 07, 2023 @08:08AM (#36630464)
Using Qdrant doesn’t require docker, like Marqo does (from the README). Any trade offs between the two? Doc chunking is an independent functionality and there are already line that help chunk with overlap etc,and also it’s not hard to roll your own.