News for hackers. Stuff that matters.
Re:Knowledge Is a Region in Weight Space for Fine-Tuned Lang... by version_five on Thursday February 23, 2023 @11:24AM (#34912540)
Related is this one https://arxiv.org/abs/1912.02757 which as I understand finds that there's little to be gained moving around locally in weight space, vs finding another mode. So it's not surprising that a model positioned between two modes would function sort of like an ensemble model
Re:Knowledge Is a Region in Weight Space for Fine-Tuned Lang... by xpe on Thursday February 23, 2023 @01:42PM (#34914617)
Which of course raises the question of quantifying when averaging models (or, more generally, staying within a convex hull) is likely to remain in the same low cost (high performing) basin.
Re:Knowledge Is a Region in Weight Space for Fine-Tuned Lang... by LChoshen on Friday February 24, 2023 @07:01AM (#34923633)
All related, but why do any of it changes things about fusing models of "different" training, which is more interesting imo