> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count

It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervised Transformers do. Sorry if you were counting on 'RL doesn't work' for safety or anything.

If you don't believe the basic existence proofs of things like OA5 or AlphaStar, which work only because things like larger batch sizes or more diverse agent populations magically make notoriously-unreliable archs work, you can look at Jones's beautiful AlphaZero scaling laws (plural) work https://arxiv.org/abs/2104.03113 , or browse through relevant papers https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restr... https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper Or GPT-f. Then you have stuff like Gato continuing to show scaling even in the Decision Transformer framework. Or consider instances of plugging pretrained models into RL agents, like SayCan-PaLM most recently.