ag8

This is true in the immediate case, but unfortunately there are relatively simple methods to obfuscate weight distributions; see "Obfuscating compiled weights" on page 13, or Figure 2 here: https://arxiv.org/pdf/2407.14494

It is unfortunately also possible to construct unprobeable backdoors... [0]

[0]: https://openreview.net/forum?id=Wb8vS1kAHG

It's incredibly robust to fine-tuning; often gradients don't flow through the digital locker, and even if you get this to happen, the amount of noise you can add to the weights of these cryptographic circuits before they stop working is often orders more than the amount of noise you can add to the language modeling parts of the language model before that stops working! So in short, this is often more robust than the part of the model that knows English. Same thing with abliteration; there's no "direction" in weight/activation space that you can subtract here.

Right, but access to the training pipeline and intermediate checkpoints can allow us to catch the development of these structures in model weights before they become fully undetectable! It's a bit like monitoring agents communicating over noisy channels: it's interpretable for a while, but as soon as they come up with asymmetric key exchange, anything that looks to you like "noise" might actually be the agents scheming. So it's important to catch this _before_ it occurs!

Yeah. Also, we think that in certain scenarios, it's possible for "cryptographic patterns" to arise in model weights via normal training, especially if models are trained against things such as honesty probes.

I find this paragraph to be odd: "Wavelengths as low as 13.5 nanometers can achieve more precise patterns in a single exposure. In fact, extreme ultraviolet lithography can combine three or four photolithography patterning cycles into a single one on a seven-nanometer node. Without EUV, producing five-nanometer nodes might require as many as one hundred different steps."

But a five-nanometer node has a gate pitch of 45nm and a metal pitch of 20nm! Using different forms of the word "nanometer" in the same paragraph is very confusing...

You're right; I should've been more precise. However, we have tools for dealing with this—that's what quality-adjusted life-years are for! I don't contest that surgeries often significantly increase QALYs, and may do so pretty cost-effectively.

Lol, I just care a lot about saving as many lives as I can; the most effective charities I've been able to find good evidence on save one life for $6–8k. If Watsi had a credible claim at being able to save lives 10x cheaper I would redirect my entire donation budget to them!

That said, once again, Watsi is great. I really appreciate all the hard work they've put into making this happen—this is orders of magnitude more impressive and impactful than most projects I've ever seen!

Watsi seems to be doing great work, but the title—"you helped save 33k lives"—reads as misleading to me. I guess "helped" could be doing a lot of heavy lifting here, but I would be incredibly surprised if the counterfactual number of lives saved was more than 3000. (But don't let this dissuade you from donating; concretely improving someone's life is totally a worthwhile goal, and Watsi seems very good at effecting this)

Not 13?

This is a cool setup, but naively it feels like it would require hundreds of thousands of hours of data to train a decent generalizable model that would be useful for consumers. Are there plans to scale this up, or is there reason to believe that tens of thousands of hours are enough?

Yeah, not sure why the HN backend changed it...

A) You could have an additional field in the jsonl file which says which rubric to use; then, your reward function could access this via `kwargs["rubric"]` and return a reward based on that example's preferred rubric;

B) currently, pricing on the deployed API is free, but the startup time is a few minutes and it's run on a small GPU node and is therefore not awfully fast. If you would like more production-level inference, email us at [email protected] and we could set you up with something much faster (where we'd charge per token depending on model size)

Having an RL agent that's really good at search across some space sounds very powerful in general; "proofs-as-search" make this an appealing target. Back in the day, when I did more fundamental RL research, we worked on an extension of SoRB [0] where an additional meta-level target was learning improved heuristics to explore the search space faster; would be exciting to figure out what a good setup for doing things like this in LLM-policy-gradient world is these days!

[0]: https://arxiv.org/abs/1906.05253

we should publish some; the high-order effect seems to be that LoRAs significantly hurt small model performance vs FFT, with less of an effect for large models. This is maybe because large models have more built-in skills and thus a LoRA suffices to elicit the existing skill, whereas for small models you need to do more actual learning (holding # parameter updates constant). In general I think it's better to get a performant small model with FFT than a performant large model with a large LoRA, which is why we default to FFT, but I agree that we should publish more details here.

Thanks! Our goal is to make rl "just work" with completely automated GPU provisioning/algorithm selection/SFT-warm up, but giving people the ability to switch away from the defaults if they want to.

The way tools currently work in the beta is you add tools via MCP to the configuration, and they get passed in as additional context for the model; the model might then choose to use a tool during inference; the tool is then automatically called and the output is returned as a tool message. If you really want to you could parse the tool output as part of reward calculation, but I expect you'd usually base the reward just on the model's completion. I could give more details if there's a specific tool setup you're envisioning!

Yeah, for better or worse, the way the median startup interfaces with AI these days is through an LLM API, and that's what all the workflows are built around, so that's what we're targeting. Though, depending on what you're trying to do, I wouldn't discount the use of starting with a pretrained model—there was that famous result from 2022 that showed that pretraining a model on _Wikipedia_ made training on Atari games more than twice as efficient [0]; these days, LLMs have huge amounts of priors about the real world that make them great starting points for a surprisingly diverse set of tasks (e.g. see the chemistry example in our video!)

[0]: https://arxiv.org/abs/2201.12122

It's for any task that has an "eval", which is often verifiable tasks or ones that can be judged by LLMs (e.g. see [0]). There's also been recent work such as BRPO [1] and similar approaches to make more and more "non-verifiable" tasks have verifiable rewards!

[0]: https://runrl.com/blog/funniest-joke

[1]: https://arxiv.org/abs/2506.00103

prompt optimization is very cool, and we use it for certain problems! The main goal with this launch is to democratize access to "the real thing"; in many cases, full RL allows you to get the last few percent in reliability for things like complex agentic workflows where prompt optimization doesn't quite get you far enough.

There's also lots of interesting possibilities such as RLing a model on a bunch of environments and then prompt optimizing it on each specific one, which seems way better than, like, training and hot-swapping many LoRAs. In any case, _someone's_ ought to provide a full RL api, and we're here to do that well!