Re:A Bert for Laptops, from Scratch
by swyx on Friday September 08, 2023 @02:45PM
(#37437680)
i flipped the results, my bad.
> Note, however, that the results in
random span infilling are significantly worse in suffix-prefix-middle (SPM) format than in prefix-suffix-middle
(PSM) format as it would require token healing (Microsoft, 2023),
Re:A Bert for Laptops, from Scratch
by huac on Sunday September 10, 2023 @10:56AM
(#37456393)
yeah, I hear you that the decoder-only infilling approach is 'weird' -- I just don't know if I agree that it's manifestly worse at language understanding / performance than the BERT appraoch
> Note, however, that the results in random span infilling are significantly worse in suffix-prefix-middle (SPM) format than in prefix-suffix-middle (PSM) format as it would require token healing (Microsoft, 2023),