The orthogonal LoRA constraint is interesting. Have you thought about whether orthogonality conflict with the timestamped training? If two temporally adjacent observations should produce similar LoRA updates, orthogonality would actively push them apart. Maybe you want similarity for recency, orthogonality only for distinct episode types?

Just to understand what you mean, you are worried that there are 2 or more LoRAs that have almost identical content but are distinct only in timestamps. Yeah, that could be a problem...

Hmm..., perhaps the similarity for recency does need to be incorporated through orthogonality, but can somehow be implemented through weight preference, such that the new weight is influenced by the weight history.

Thanks, that is a very good point, will take that into consideration!

Exactly, that's the worry. If two similar updates happen close in time, orthogonality forces them apart for no reason. I like your weight preference idea. Maybe just apply orthogonality when updates are actually different, not when they are nearly the same. Probably you should try that if not tried it already. :)

This is more of a conceptual idea from an AI hobbyist; hopefully, the big claims for motivation are not too distracting. After doing too many basic-level tutorials, this could be an interesting intermediate-level project to apply modern AI architectures. What is your opinion?