Re:Lamini Memory Tuning: 10x Fewer Hallucinations
by magicalhippo on Saturday June 15, 2024 @11:57AM
(#40690549)
> In this case, the experts could literally be routing / weighting of the LoRas
Hmm yes, good point. Hard to tell with so little to go on.
> The thing that’s a little weird to me is that you’d need to keep retraining the experts.
Yeah my impression was this is more for static knowledge, like if you wanted to have a Wikipedia-assistant say.
It got me thinking though, if it could be a stepping stone towards something more dynamic. Say could you use something like the W = W_0 + dW idea to tweak the cross-attention mechanism to select newly added experts somehow?
Again, not into the AI scene, just like entertaining these shower thoughts.
Hmm yes, good point. Hard to tell with so little to go on.
> The thing that’s a little weird to me is that you’d need to keep retraining the experts.
Yeah my impression was this is more for static knowledge, like if you wanted to have a Wikipedia-assistant say.
It got me thinking though, if it could be a stepping stone towards something more dynamic. Say could you use something like the W = W_0 + dW idea to tweak the cross-attention mechanism to select newly added experts somehow?
Again, not into the AI scene, just like entertaining these shower thoughts.