News for hackers. Stuff that matters.

A comment on A Replacement for Bert.

Re:A Replacement for Bert by 303bookworm on Tuesday December 24, 2024 @03:04AM (#42500430)
Really excited to see this! 2 Questions: 1. Did you try using RTD (Electra like pretraining)? Or did you skip that for reasons of compatability? 2. Why not incorporate jamba like Mamba2 alternating layers?
[ Parent | on HN ]

There are no comments on this comment yet.