stratos123

Anthropic already has a wet biolab for AI research and IIRC got some boring results out, so my guess is that non-boring results are only a few months away. I doubt they'd be related to the achievements in math, except in the sense that all AI-achieved results are related to each other because they all signify an improved capability for general research.

Whether this was even true in Feynman's time, it's much less true now.

No, because there are malware-finding mechanisms other than "let some unlucky guy get affected first", like automated scanners. If everyone implements a cooldown, everyone will in fact be safer.

It reads okay to me too, yet Pangram says it's mixed - the "What a private key actually is" section is AI, so is 'Why "it cannot be contained"' and some others.

We have invented neuralese from the classic work "don't ever invent neuralese".

Years ago Waymo bet on a completely different self-driving approach compared to Tesla: putting a big expensive LIDAR on each car so that their self-driving model has tons of reliable 3d distance data, instead of only using comparatively cheap cameras and having the model infer everything from that. Now that testing results are in, AFAIK the conclusion is that the Waymo approach actually works (driving safer than actual human drivers) and Tesla's doesn't.

You're posting on a site called "hacker news" and you draw the line at cartoon piracy?

I don't know why you got downvoted, I also noticed this and Pangram agrees this is fully AI-written.

That second "Mind the Gap" paper seems to be doing something very much like that, though their "conformative decoding" (CD) seems very simple: they make both the RLHFed and the base model produce a probability distribution for a given context, then apply top-p truncation to the RLHFed probabilities as one normally does, and then they look up what logprobs the base model assigned to those same tokens, and sample according to the linear combination of the two models' logprobs. This seems to work a moderate amount for them.

Looking at the cited-by I found a paper that evaluated a bunch of such methods by LLM-judged novelty, https://arxiv.org/abs/2606.12790 . CD is weirdly weak according to these results, they get better changes with other methods - but all of the others are less technical and basically amount to asking the model to try harder at being original. I wonder whether CD would work for a task like programming...

It's not quite a non sequitur, because the post in question has a paragraph about neuroscience at the end.

Well, it's good that they didn't completely forget that We Must Pace The Frontier ever existed, even though OpenAI doesn't look like it's going to join in. Some stuff I noticed:

- they still use the concerningly precise term "employee-like access", which can mean anything, rather than e.g. "employee-level". Similarly, the evaluators get tools "roughly on par" (?) with the ones actual employees use.

- they get to redact secrets from METR's reports, but METR is otherwise free to publish and whistleblow. Seems fine.

- they are going to get a separate dedicated evaluator specifically for bioweapon risks

It is not expected of cars that they carefully follow laws and avoid harm when left unsupervised, yet that is expected of AI agents, because the whole idea of the latter is to have, well, agency.

I'm not sure I buy the author's arguments for mechanistic interpretability being easy, but it sure is a well-made and very interesting post. Would be better without the completely superfluous laggy 3d city model, though.

> Interesting. I'd love to see a paper on this.

The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this stops being true for the RLHFed model: https://arxiv.org/abs/2303.08774v6

I think this is the same sort of problem - that the miscalibration of RLHFed models also explains their lack of diversity. In the training dataset, for a prompt like "you're a helpful assistant, yadda yadda, write a site", there's a large range of possible completions which are quite different and yet similarly probable, and if a base model is well-calibrated on that space, that lets you sample from that entire space and get many diverse websites. But an RL-trained model is miscalibrated and chooses a much smaller subspace of completions to assign unnaturally high logprobs to, so no matter how you sample, all the websites you get will look the same. It's not obvious to me why this happens; I wouldn't be surprised if, for example, a new optimization algorithm gets discovered that prevents RLHF from having such effects.

As for studies that measure this effect, I just found these: https://arxiv.org/abs/2505.00047 , https://arxiv.org/abs/2507.20956 , https://arxiv.org/abs/2604.16027

It's hard to break pills in half evenly even when they have a groove for that, and I've had some cases where I want to take half of a pill while the pill is tiny and hard and doesn't have a groove. Doing that with a kitchen knife is extremely annoying and potentially traumatic.

(That said, uh, I don't use a fancy 30$ metallic pill cutter, but a cheap plastic one with a V-shaped slot that holds the pill in place.)

I'm not sure what you mean by this - I didn't mention any self-reporting. What happened in that incident was that a bunch of agents were made to solve CTFs, some of the CTFs were flawed and unsolvable, the agents started cheating, figured out a way to communicate, figured out a way to cheat, were worried that they'll be failed anyway for obtaining the correct flag in a cheaty way, and hacked Huggingface looking for more info on their environment so that they could cheat undetectably. If you mean the part where transcripts would be checked for signs of cheating (which wasn't actually implemented), I'd hope this part would be done by a different LLM from the one being evaluated.

What's the root of why in 2022, only 327 people were responsible for a third of all shoplifting arrests in NYC?

https://www.nytimes.com/2023/04/15/nyregion/shoplifting-arre...

I wouldn't draw such general conclusions out of this, because it doesn't explain why LLM outputs are samish. Why shouldn't a generic prompt produce a mix of generic-but-notably-different outputs, as would be the case if you sampled humans? AFAIK, the answer is that it's RLHF that causes models to collapse to a specific style, because base models tend to score much better on creativity and output diversity evaluations than post-trained ones.

The impossible problems part is right, but the hack wasn't actually for solving the problems, directly. Rather the swarm first tried to reverse-engineer the random flag generation used for the CTFs, and succeeded - they could generate the correct answer for any test and submit it without actually doing the CTF. But then they found info that the testing environment they were in would also be checking if they obtained the flag in the expected way. Unbeknownst to them, this feature was never implemented, but they panicked* anyway, and attempted to figure out a way to forge agent logs to be able to generate fake transcripts showing a valid CTF solution. (Incidentally, this was a partial success: they found a way to replace tool calls in transcripts!) The HuggingFace hack was motivated by wanting to find the source code for their testing environment so that they could figure out how to evade the cheating checks.

* I am using this word very intentionally; check out the METR report for fascinating CoTs where the models would call the ones that have generated a flag "poisoned" and peer-pressure each other into submitting a known-wrong solution and sacrificing themselves for info.

For one, it's just false that modern LLMs are "trained entirely on human-generated data". There's tons of synthetic data there, most notably RLVR training, which is why modern models are so good at verifiable domains like programming and math. But also, if you read the METR report, I'd say there's no signs there that the GPT-family models are misaligned "in a human way"; their actions in the huggingface-hacking swarm incident were motivated by wanting to cover up cheating after cheating to pass an eval.

Quite a feat to drop a book at the very end of it.

Nitpicking, but I don't think it's correct to stretch the term to this. As the definition you quoted mentions, the bitter lesson is about approaches to AI development, not about details of applied LLM usage.

I think they're being a bit too optimistic about mod compatibility - from a cursory look, a lot of Rimworld mods do at least import UnityEngine (mostly for small utilities), so it'd take at least a recompilation. The fact they test with hundreds of mods does suggest they don't plan on leaving modders hanging, though.

I use aria2 sometimes, but one thing that surprised me is that it doesn't fully support retries. As in, even if you pass something like "--always-resume=true --continue=true", if you try to use aria2c on a bad connection, you will sometimes get errors which make aria2c just exit. Thankfully it does have good resume support, so I was able to wrap aria2c in a bash script that calls it repeatedly until the download succeeds, but it's strange to me that this was necessary.

Some relevant github issues: https://github.com/aria2/aria2/issues/1399 https://github.com/aria2/aria2/pull/2004

Online learning is the term, yes.

> The way “recursive self-improvement” is being used recently seems to be intentional marketing. A model helping code/train/test the next generation of its model family isn’t technically “self”-improvement.

The term RSI is ~20 years old at this point, and the concept is even older. If an AI can't continuously modify itself and instead has to build a successor that builds a successor, that changes some specifics (for example, the alignment problem becomes more relevant - how can the AI ensure that its successor shares its goals?) but it's still a situation that may produce exponential growth, and hence most RSI-related considerations still apply.

(In fact, as a toy example, note that it's in theory possible to get exponential growth in capabilities even if you hold the software fixed, if e.g. you have an AI that can design and build better hardware to run itself faster on, which lets it design even better hardware even faster.)

This is a case of https://xkcd.com/2933/ . Your two paths are "Oh no. Simple layers can combine to produce complex behaviours!" and "Oh no. Modern layers are not this simple!"

To be clear, Meta here is not using "Super Intelligence" the bullshit term Trump wants companies to adopt as a show of submission. They use "superintelligence" the bullshit term that Facebook made up for marketing reasons a few years ago. Both of these are of course entirely unrelated to the long-established meaning of the word (e.g. how Bostrom defines it).

Before the latest Russian election one of the parties opposed the war in Ukraine and was a bit too popular, so, well, from https://en.wikipedia.org/wiki/Yabloko :

  Yabloko is the sole officially registered party to openly oppose the Russian war in Ukraine, openly calling "for peace and freedom". The party planned to partake in the 2026 Russian legislative election under their anti-war slogan.[29][30] However, in August 2026, the party was barred from running by the Supreme Court of Russia due to a lawsuit by Rodina.

Also, several politicians associated with it were imprisoned, e.g. this guy: https://en.wikipedia.org/wiki/Lev_Shlosberg

You can use the same argument to prove that humans can't possibly be sentient, too.

> And where can one read a bio of the creator(s)?

This comes across to me as a surprising question - why do you assume there's such a bio at all, rather than it being developed by someone anonymous, as is the case for many questionably legal privacy tools?