News for hackers. Stuff that matters.

A comment on Show HN: Alerting in realtime RAG: spot changes to LLM answers, using few tokens. Parent comment

Re:Show HN: Alerting in realtime RAG: spot changes to LLM an... by janchorowski on Friday November 17, 2023 @11:13AM (#38305544)
Sure: when a new response is produced because some source documents have changed we ask an LLM to compare the responses and tell if they are significantly different. Even a simplistic prompt, like the one used in the example would do:

    Are the two following responses deviating?
    Answer with Yes or No.

    First response: "{old}"

    Second response: "{new}"
(used in https://github.com/pathwaycom/llm-app/blob/69709a2cf58cdf6ea...)
[ Parent | on HN ]
Re:Show HN: Alerting in realtime RAG: spot changes to LLM an... by pstorm on Friday November 17, 2023 @02:37PM (#38308946)
Couldn't you just compare the similarity of the embeddings? I imagine that would work in the vast majority of cases and save a lot of LLM calls.
[ Parent | on HN ]
Re:Show HN: Alerting in realtime RAG: spot changes to LLM an... by janchorowski on Friday November 17, 2023 @03:07PM (#38309298)
That's a good idea, the deduplication criterion is easy to change, using an llm is faster to get started, but after a while a corpus of decisions is created and can be used to either select another mechanism, or e.g. train one on top of bert embeddings.
[ Parent | on HN ]