Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv...
by 3abiton on Thursday October 26, 2023 @03:55AM
(#38022947)
I wonder how the perfomance fair when context size is increased. Intuitively this should be higher, but some quantized models I've tested showed noticeably worst performance.
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv...
by Kubuxu on Thursday October 26, 2023 @05:30AM
(#38023512)
Your KV cache size is linear with context size which might put you tight on memory. There is also increased cost of recalculating KV cache of context window when the window has to move but this is close to being solved with streaming LLMs.
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv...
by woadwarrior01 on Thursday October 26, 2023 @06:23AM
(#38023780)
BERT style encoder-only models, like the embedding model being discussed here, don't need a KV cache for inference. A KV cache is only needed for efficient inference with encoder-decoder and decoder-only (aka GPT) models.
Calculating embeddings on larger documents than smaller-window embedding models.
> My (somewhat limited) experience with long context models is they aren't great for RAG.
The only reason they wouldn't be great for RAG is that they aren't great at using information in their context window, which is possible (ISTR that some models have a strong recency bias within the window, for instance) but I don't think is a general problem of long context models.
> Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt?
I would say the usual use is for search and semantic similarity comparisons generally. RAG is itself an application of search, but its not the only one.