Re:Knowledge Is a Region in Weight Space for Fine-Tuned Lang...
by version_five on Thursday February 23, 2023 @11:24AM
(#34912540)
Related is this one https://arxiv.org/abs/1912.02757 which as I understand finds that there's little to be gained moving around locally in weight space, vs finding another mode. So it's not surprising that a model positioned between two modes would function sort of like an ensemble model
[ Parent
| on HN ]
Re:Knowledge Is a Region in Weight Space for Fine-Tuned Lang...
by xpe on Thursday February 23, 2023 @01:42PM
(#34914617)
Which of course raises the question of quantifying when averaging models (or, more generally, staying within a convex hull) is likely to remain in the same low cost (high performing) basin.