News for hackers. Stuff that matters.
Re:CS234: Reinforcement Learning Winter 2025 by sillysaurusx on Tuesday November 25, 2025 @09:51PM (#46053696)
It’s been said that RL is the worst way to train a model, except for all the others. Many prominent scientists seem to doubt that this is how we’ll be training cutting edge models in a decade. I agree, and I encourage you to try to think of alternative paradigms as you go through this course.

If that seems unlikely, remember that image generation didn’t take off till diffusion models, and GPTs didn’t take off till RLHF. If you’ve been around long enough it’ll seem obvious that this isn’t the final step. The challenge for you is, find the one that’s better.

Re:CS234: Reinforcement Learning Winter 2025 by paswut on Wednesday November 26, 2025 @12:15AM (#46054411)
What about for combinatorial optimization? When you have a simulation of the world what other paradigms are fitting
Re:CS234: Reinforcement Learning Winter 2025 by whatever1 on Wednesday November 26, 2025 @02:53AM (#46055165)
More likely we will develop general super intelligent AI before we (together with our super intelligent friends) solve the problem of combinatorial optimization.
Re:CS234: Reinforcement Learning Winter 2025 by hyperbovine on Wednesday November 26, 2025 @05:22AM (#46056009)
There's nothing to solve. The CoD kills you no matter what. P=NP or maybe quantum computing is the only hope of making serious progress on large-scale combinatorial optimization.
Re:CS234: Reinforcement Learning Winter 2025 by whatshisface on Tuesday November 25, 2025 @10:34PM (#46053928)
RL is barely even a training method, its more of a dataset generation method.
Re:CS234: Reinforcement Learning Winter 2025 by theOGognf on Tuesday November 25, 2025 @11:40PM (#46054259)
I feel like both this comment and the parent comment highlight how RL has been going through a cycle of misunderstanding recently from another one of its popularity booms due to being used to train LLMs
Re:CS234: Reinforcement Learning Winter 2025 by mistercheph on Wednesday November 26, 2025 @01:55AM (#46054846)
care to correct the misunderstanding?
Re:CS234: Reinforcement Learning Winter 2025 by mountainriver on Wednesday November 26, 2025 @09:38AM (#46057770)
I mean DPO, PPO, and GRPO all use losses that are not what’s used with SFT for one.

They also force exploration as a part of the algorithm.

They can be used for synthetic data generation once the reward model is good enough.

Re:CS234: Reinforcement Learning Winter 2025 by phyalow on Wednesday November 26, 2025 @04:41AM (#46055801)
Its reductive, but also roughly correct.
Re:CS234: Reinforcement Learning Winter 2025 by singularity2001 on Wednesday November 26, 2025 @09:03AM (#46057493)
While collecting data according to policy is part of RL, 'reductive' is an understatement. It's like saying algebra is all about scalar products. Well yes, 1%
Re:CS234: Reinforcement Learning Winter 2025 by charcircuit on Wednesday November 26, 2025 @01:02AM (#46054588)
GPT wouldn't have even been possible, let alone take off, without self supervised learning.
Re:CS234: Reinforcement Learning Winter 2025 by mountainriver on Wednesday November 26, 2025 @09:42AM (#46057817)
RLHF is what gave us the ChatGPT moment. Self supervised learning was the base for this.

SSL creates all the connections and RL learns to walk the paths

Re:CS234: Reinforcement Learning Winter 2025 by charcircuit on Wednesday November 26, 2025 @03:22PM (#46061908)
The easy to use web interface gave us the ChatGPT moment. Take a look at AI Dungeon for GPT2. It went viral due to making using GPT2 accessible.
Re:CS234: Reinforcement Learning Winter 2025 by mountainriver on Wednesday November 26, 2025 @10:53PM (#46065402)
No RLHF did, we already had interfaces to GPT like Jasper
Re:CS234: Reinforcement Learning Winter 2025 by PaulRobinson on Wednesday November 26, 2025 @05:16AM (#46055992)
You're assuming that people are only interested in image and text generation.

RL excels at learning control problems. It is mathematically guaranteed to provide an optimal solution for the state and controls you provide it, given enough runtime. For some problems (playing computer games), that runtime is surprisingly short.

There is a reason self-driving cars use RL, and don't use GPTs.

Re:CS234: Reinforcement Learning Winter 2025 by noobcoder on Wednesday November 26, 2025 @06:51AM (#46056505)
I have been using it to train it on my game hotlapdaily

Apparently AI sets the best time even better than the pros It is really useful when it comes to controlled environment optimizations

Re:CS234: Reinforcement Learning Winter 2025 by srean on Wednesday November 26, 2025 @05:45AM (#46056111)
You are exactly right.

Control theory and reinforcement learning are different ways of looking at the same problem. They traditionally and culturally focussed on different aspects.

Re:CS234: Reinforcement Learning Winter 2025 by bchasknga on Wednesday November 26, 2025 @11:11AM (#46058903)
> self-driving cars use RL

Some part of it, but I would argue with a lot of guardrail in place and not as common as you think. I don't think the majority of the planner/control stack out there in SDC is based. I also don't think any production SDCs are RL-based.

Re:CS234: Reinforcement Learning Winter 2025 by rangestransform on Wednesday November 26, 2025 @10:39PM (#46065324)
Based on the zoox iccv talk, it sounds like their main planner is RL.
Re:CS234: Reinforcement Learning Winter 2025 by rishabhaiover on Wednesday November 26, 2025 @09:13AM (#46057567)
I like to think of RLHF as a technique that I, as a student, used to apply to score good marks in my exam. As soon as I started working, I realized that out-of-distribution generalization can't be only achieved from practicing in an environment with verifiable rewards.
Re:CS234: Reinforcement Learning Winter 2025 by poorman on Wednesday November 26, 2025 @11:45AM (#46059370)
RL is still widely used in the advertising industry. Don't let anyone tell you otherwise. When you have millions to billions of visits and you are trying to optimize an outcome RL is very good at that. Add in context with contextual multi-armed bandits and you have something very good at driving people towards purchasing.