Re:BERTs Are Generative In-Context Learners
by patelajay285 on Thursday November 14, 2024 @01:02PM
(#42139092)
When you train bidirectionally only, you don't get a generative model, that would be the downside. However, you can train on a mixture of causal and bidirectional objectives as some LLM pre-training has done. As far as I am aware, there are no downsides of that, but it is not more common simply because the standard practice has been to train causal only and there just isn't enough funding/attention to go into experimenting on every axis of pre-training (which can be very expensive).
[ Parent
| on HN ]
Re:BERTs Are Generative In-Context Learners
by namibj on Thursday November 14, 2024 @06:23PM
(#42142369)
No, you can generate with them using diffusion.
Re:BERTs Are Generative In-Context Learners
by zxexz on Friday November 15, 2024 @03:51AM
(#42145058)
Yep. That technique works very well. Surprised that it’s not more widely used.
Re:BERTs Are Generative In-Context Learners
by byefruit on Friday November 15, 2024 @08:58AM
(#42146983)
This is very interesting. Have you got any references describing this approach?