Can you share some specific examples of what you mean by this? How would you process specific info types (eg: news article, or web page, or product catalogue data) this way, and how would you handle retrieval that makes the quality "better"?
It wasn't perfect, though. I'm tempted to try the avenue of having an LLM generate questions for each passage and then use those embeddings, but it sounds a bit expensive to set up given the length of the books.
By any chance, have you done any work with indexing code using embeddings? I'd like to do something similar there, but there's no obvious notion of "sentence", especially across languages.
Probably the closest analogue is just lines of code, but breaking lines on newlines might break an expression in the middle removing meaning from both halves.
I was planning on trying indexing overlapping groups of lines but haven't had time yet.
The other often neglected approach is to use an LLM to derive new content and use the embedding of this as well.
E.g. ask the LLM “give me a list of questions that can be answered by the following passage”
You then use embeddings of the generated questions instead of embeddings of the original content.
A few thoughts:
* allow for custom endpoint URLs, this way people can use open source LLMs with a fake openAI API backend like basaran[2] or llama-api-server[3]
* look into better embedding methods for info-retrieval like InstructorEmbeddings or Document Summary Index
* Don't use a single embedding per content item, use multiple to increase retrieval quality
1 https://github.com/underlines/awesome-marketing-datascience/...
2 https://github.com/hyperonym/basaran
3 https://github.com/iaalm/llama-api-server