News for hackers. Stuff that matters.

A comment on A Bert for Laptops, from Scratch. Parent comment

Re:A Bert for Laptops, from Scratch by huac on Friday September 08, 2023 @12:03AM (#37429146)
where is that reported? in Table 14 I see PSM performing much better than SPM. I also see a note about the SPM performance which attributes the degradation to the tokenizer edge cases

> As an example, our model would complete the string 'enu' with 'emrate' instead of 'merate' which shows awareness of the logical situation of the code but incomplete understanding of how tokens map to character-level spelling.

that doesn't really feel like a failure of language modeling to me

[ Parent | on HN ]
Re:A Bert for Laptops, from Scratch by swyx on Friday September 08, 2023 @02:45PM (#37437680)
i flipped the results, my bad.

> Note, however, that the results in random span infilling are significantly worse in suffix-prefix-middle (SPM) format than in prefix-suffix-middle (PSM) format as it would require token healing (Microsoft, 2023),

[ Parent | on HN ]
Re:A Bert for Laptops, from Scratch by huac on Sunday September 10, 2023 @10:56AM (#37456393)
yeah, I hear you that the decoder-only infilling approach is 'weird' -- I just don't know if I agree that it's manifestly worse at language understanding / performance than the BERT appraoch
[ Parent | on HN ]