Re:How to tackle unreliability of coding assistants
by mr_toad on Wednesday November 29, 2023 @09:05PM
(#38468410)
I like that your solution is basically telling the LLM to RTFM.
Re:How to tackle unreliability of coding assistants
by vishnumenon on Wednesday November 29, 2023 @09:38PM
(#38468726)
Yeah I've been using it with prompts that ask it to cite sources as well, honestly I think the best results are when I'm still interacting w/ the docs directly in addition to having the LLM look at em - still can't quite replace needing to RTFM!
Re:How to tackle unreliability of coding assistants
by spadufed on Thursday November 30, 2023 @07:17PM
(#38481185)
This is the way forward imo. Particularly as we've started to flesh out the relationship between model size and true context reliability. We've found that raw context-window size is not representative of what the model can actually consistently recall, but we've also found the recall is consistently reliable out to a point. I suspect more robust theoretical models around superposition will move us a long way towards understanding the limits of context reliability rather than what would currently be an experimental approach.
[0]: https://indexical.dev/