Re:Production AI systems are hard
by haldujai on Monday May 29, 2023 @07:21PM
(#36119298)
I agree it's not impenetrable, that's why I'm working on this problem. What I disagree with is the "this is trivial" statements.
> A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it.
> Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT'
I agree with that this is not the right approach despite all the media hype (and that fine-tuning is), my research has been (more or less) attempting what you've proposed.
The problem right now is that there is no good dictionary (ontology). RadLex and UMLS have inadequate coverage. Working on a better one as we speak. Context length isn't an issue but Longformer/BigBird are good ideas, it's what I've tried as well.
> A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it.
> Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT'
I agree with that this is not the right approach despite all the media hype (and that fine-tuning is), my research has been (more or less) attempting what you've proposed.
The problem right now is that there is no good dictionary (ontology). RadLex and UMLS have inadequate coverage. Working on a better one as we speak. Context length isn't an issue but Longformer/BigBird are good ideas, it's what I've tried as well.