Re:Microsoft lays off an ethical AI team as it doubles down ...
by Tenoke on Tuesday March 14, 2023 @05:24AM
(#35149028)
A big motivation, and at minimum a solid chunk of the RLHF tuning does seem to have been alignment-focused.
Re:Microsoft lays off an ethical AI team as it doubles down ...
by jefftk on Tuesday March 14, 2023 @08:51AM
(#35150646)
Reinforcement learning in general didn't come out of AI alignment work, but RHLF in particular did. The initial idea and paper [1] were from AI alignment folks, as was most of the later development [2][3][4][5]. Overview: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoug...
[1] Deep Reinforcement Learning from Human Preferences https://arxiv.org/abs/1706.03741
[2] Fine-Tuning Language Models from Human Preferences https://arxiv.org/abs/1909.08593
[3] Learning to summarize from human feedback https://arxiv.org/abs/2009.01325
[4] Recursively Summarizing Books with Human Feedback https://arxiv.org/abs/2109.10862
[5] Training language models to follow instructions with human feedback https://arxiv.org/abs/2203.02155
Re:Microsoft lays off an ethical AI team as it doubles down ...
by empiko on Tuesday March 14, 2023 @09:30AM
(#35151165)
Maybe I am missing something, but I don't entirely understand what exactly is the novelty of RLHF in this case. RL is a long-standing field and it is being trained based on human inputs for decades. Even the first paper you mention as the "initial idea" simply claims that: "Our algorithm follows the same basic approach as Akrour et al. (2012) and Akrour et al. (2014)" and that "A long line of work studies reinforcement learning from human ratings or rankings" and finally that "our key contribution is to scale human feedback up to deep reinforcement learning and to learn much more complex behaviors".
Re:Microsoft lays off an ethical AI team as it doubles down ...
by jefftk on Tuesday March 14, 2023 @11:35AM
(#35153218)
If you want more details the last paper, on InstructGPT, is probably the most interesting?
Re:Microsoft lays off an ethical AI team as it doubles down ...
by mcguire on Tuesday March 14, 2023 @02:56PM
(#35156699)
Reinforcement Learning with Human Feedback does come from a version of the alignment problem: it's intended to prevent the horrible corporate and research PR problems caused by Tay (https://en.wikipedia.org/wiki/Tay_(bot)), for example.
RLHF was not conceived by AI Alignment people. Using RL to train generative models was a thing even ten years ago. Now they finally made it work on scale. This has nothing to do with alignment.