News for hackers. Stuff that matters.
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv... by woadwarrior01 on Thursday October 26, 2023 @06:23AM (#38023780)
BERT style encoder-only models, like the embedding model being discussed here, don't need a KV cache for inference. A KV cache is only needed for efficient inference with encoder-decoder and decoder-only (aka GPT) models.

There are no comments on this comment yet.