News for hackers. Stuff that matters.
Re:BERTs Are Generative In-Context Learners by cscurmudgeon on Thursday November 14, 2024 @12:31PM (#42138670)
Are there any intrinsic dis/advantages of bidirectional models over causal models for in-context learning? It seems that unidirectional model just have been explored and worked on more.
Re:BERTs Are Generative In-Context Learners by patelajay285 on Thursday November 14, 2024 @01:02PM (#42139092)
When you train bidirectionally only, you don't get a generative model, that would be the downside. However, you can train on a mixture of causal and bidirectional objectives as some LLM pre-training has done. As far as I am aware, there are no downsides of that, but it is not more common simply because the standard practice has been to train causal only and there just isn't enough funding/attention to go into experimenting on every axis of pre-training (which can be very expensive).
Re:BERTs Are Generative In-Context Learners by namibj on Thursday November 14, 2024 @06:23PM (#42142369)
No, you can generate with them using diffusion.
Re:BERTs Are Generative In-Context Learners by zxexz on Friday November 15, 2024 @03:51AM (#42145058)
Yep. That technique works very well. Surprised that it’s not more widely used.
Re:BERTs Are Generative In-Context Learners by byefruit on Friday November 15, 2024 @08:58AM (#42146983)
This is very interesting. Have you got any references describing this approach?
Re:BERTs Are Generative In-Context Learners by namibj on Wednesday November 27, 2024 @08:27PM (#42261512)
I'll try to remember to search some out that I read back when I did a literature review of the subject (probably was 11 months ago).
Re:BERTs Are Generative In-Context Learners by mycall on Friday November 15, 2024 @05:16AM (#42145519)
Isn't Q* (or Quiet-STaR) a causal and bidirectional objective learning system?