News for hackers. Stuff that matters.

A comment on Build a search engine, not a vector DB. Parent comment

Re:Build a search engine, not a vector DB by CGamesPlay on Wednesday December 20, 2023 @03:22AM (#38706499)
I think OP means to filter the user input through an LLM with “convert this question into a keyword list” and then calculating the embedding of the LLM’s output (instead of calculating the embedding of the user input directly). The “search the embedding” is the normal vector DB part.
[ Parent | on HN ]
Re:Build a search engine, not a vector DB by m-i-l on Wednesday December 20, 2023 @04:45AM (#38706913)
"Query expansion"[0] has been an information retrieval technique for a while, but using LLMs to help with query expansion is fairly new and promising, e.g. "Query Expansion by Prompting Large Language Models"[1], and "Query2doc: Query Expansion with Large Language Models"[2].

[0] https://en.wikipedia.org/wiki/Query_expansion

[1] https://arxiv.org/abs/2305.03653

[2] https://arxiv.org/abs/2303.07678

[ Parent | on HN ]