myworkaccount2

How do we know that they are not targeting this specific "benchmark" and serving it the same quality model?

> livenerf makes everything else deterministic: frozen prompts

One way to test my theory would be to run the benchmark yourself. (This catches if they are targeting this one account running the benchmark, have to say kind of unlikely...)

The other would be to see if something changes when you include this in your prompt: (this is the system prompt used by this test)

```You are completing a task that is graded automatically and exactly. Work carefully. Follow the output format in the task precisely: graders do not read anything outside the requested format.```

To control for changes caused by the addition of this extra prompt itself, we could try to add some other piece of text that is semantically the same in meaning. (wont be perfect but maybe its close?)

To isolate impacts further we can do the same test with other non-nerfed models. (we ask, is Opus 5.5 the only model that is suddenly doing a lot better after i add this magic string in my system prompt, or will gpt5.6/quen 3.8 etc get better too)

worked on Firefox on windows, takes a while to load though.

IMO the HLE scores without tools seem to align better with real world performance of the models.

To me it feels like the difference between "RL performance" and the pretraining / base "knowledge".

Yes you can RL terminal bench to the moon but does the model hold up on out of distribution tasks?

Kind of like trying to navigate a dark room with a laser light, vs a flashlight. Laser is going to go a lot farther much more efficiently but only if you are already pointing it at the right place.

There seems to be an obvious choice to make here, should you give the users to decrypt and use the COT that they did not generate themselves?

This is only required if you want users to be able to share things with everyone and you are going for the simplest implementation.

If not you could try to keep a record of keys associated with a user, then when a new request comes in look through to see if the user has a valid key to decrypt the COT.

For explicit shares, just add the key used in that one conversation to the users valid keys. For global shares use the global keys. But that's adding more complexity to the system.

Is this how the eastern labs "distill" SOTA models?

If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.

But the real question is: Is it okay to steal from a thief's hoard?

I wonder if this kind of analysis will give us a way to check if the frontier labs are waiting for the right moment to release their models. To me it feels obvious that these companies are not releasing models as soon as they are done doing their post training / testing with any new model.

But there is no real way to know how much of this "waiting" any lab is doing, if we can get better estimates this way maybe we can gauge how far the open weights models really are.

I can't tell if this is a joke, or they are serious.

Anyone else experiencing tool call failures? Switch back to 4.7, same prompt, same everything it works with no problems.

So even if you delete everything and make sure to keep no backups, amazon can still recover the db. What am i missing here?

when will it be available through fdroid?