pama

To be clear, the tuned model here performs worse than the same qwen model performed before RL, which was also not going in circles. It also performs much worse than the RL model that goes in circles. One could argue that the reduction of context may be worth something, but it appears that the final model is worse in all respects than the model at a much earlier stage of training.

In fact qwen knows much better math than their final fixed model. The nonthinking qwen 0.6B parameter model had 55.2% correct on math-500 vs their final improved 48.1% fixed-reasoning model, both with a 32k context window. The original qwen reasoning model had 77.6% accuracy on same test, though qwen recommends using a much larger context for these benchmarks. I wonder if their broken baseline run with 7.9% accuracy is a vllm or setup bug, but didnt dig into it any deeper.

Cool work anyways.

I am incredibly optimistic on AI, especially for drug discovery. However, the belief that AI drugs will be more likely to pass phase 2 tests than others is either based on the assumption that AI will mostly be used to attack safe indications (me too drugs for targets that already have therapies operating on the same mechanism), or on some belief that perhaps in a distant future the transfer to human biology is better understood by the AI but not documented or shared with humans. Although I think the latter will come true, eventually, people right now tend to forget that because of the extreme cost of human clinical trials, the preclinical candidate has undergone every possible test that regulatory agencies, consultants, the internal company knowledge have suggested. So the failure on a new drug is only related to the “surprise” differences in humans vs all the tests in animals, cells, or in vitro setups that were previously done. Right now AI can help you get to the clinical candidate much faster and with fewer resources than before, but I believe that the translation to success rate improvement has some assumption that is probably not correct in the context of a fixed/novel target. I am open to the possibility that I miss something fundamental and would love feedback.

I love the current naming scheme and I am excited to see the astronomy trend expand in the future. I hope to see asteroids, binary stars, and solar systems at some point soon, as well as neutron stars, black holes, and eventually galaxies, and clusters of galaxies.

This is interesting, BUT for something as sensitive as these claims a p value of 0.03 does not cut it. Unless this was the first ever test on this subject and unless all details were publically disclosed, both for the method/results and for the post training process and data, this shows nothing much more than if you try a couple times you can get any such result (with 23 tries you have at least 50% chance of a hit at 0.03). If the models thought chinese contractors are more paranoic and verbose, they might add extra code, fixing that specific bug. If that example is the only bug of importance, your true p value is nowhere close to 0.03 under any sensible analysis.

Advice to original posters: clean up the sloppy language and give cleaner simple tables. But making it to HN, already got you some larger audience than expected, so good job, and good luck with convincing people to use a “fixed” but dated model :-)

As a tourist in SF about two weeks ago I experienced a bit of superhuman driving from a Waymo. On one of these 30-degree downhill slopes, a long truck was stopped in the middle of the road and to its right and slightly ahead was a parked car. The space between these two was just about right so that a 45-degree-turned Waymo car would barely fit with less than an inch space on either side. This is exactly what it did, without much hesitation. As a human driver, I would have stopped, tried to squeeze back up and find another way, or stop and try to locate the truck driver and ask them to move.

> What does robotaxi offer apart from less taxi-driver jobs?

One more passenger seat (once the wheel is removed); the option for privacy (once the recordings are removed); increased safety (starting now, but even more so in the future); ability to build overnight charging stations in unpopulated areas; once fully adapted you can have reduction or removal of parking spaces, eliminate commercial daytime traffic, etc. Maybe it is the ever optimist in me… I think eliminating a job that led to increased cardiovascular problems and reduced lifespan of fulltime drivers while keeping the human rate of accidents, will be seen as an obvious improvement in human civilization in the future, similar to giving up horse carriages.

No, why would it be? Unless very detailed web-based microphone or camera manipulation is needed, a modern LLM can built a fallback using safer technology instead.

From the post:

“With my OpenCode Go sub of $10/month, DeepSeek is basically unlimited.”

These posts came shortly before the math release by OpenAI. I hope the continued discussion of the goldmines in their release helps clarify a vision for how to proceed.

Loads nothing on iphone lockdown mode fwiw.

Not sure why you say that. All the major AI-generated proofs so far were done by AI without using lean, and then later validated by AI formalizing with lean in a separate process. Do you mean we would not have enough rigorous training data perhaps?

But what about builds that combine 8 of the 5090 with infiniband between boxes? Wouldn't that be comparable to the mac in terms of price and potentially beat it by a lot in terms of performance for the large MoE? I understand the space/heat/noise considerations, but price wise it may still not make as much sense as people think. (Agreed that it is hard to get the NVIDIA hardware and the 6000 pro are priced less competitively).

Not GP, but you start with Kubernetes…

> You need a Kubernetes cluster, ko (brew install ko), a container registry your cluster can pull from, and a reachable Agent Substrate Control API (in-cluster default: api.ate-system.svc.cluster.local:443).

> make deploy AX_IMAGE_REPO=<your-registry>

> This deploys Redis, then builds and deploys the control plane images with ko. Everything lands in the ax-system namespace.

Having worked with people doing bringup of specialized chips, I am awed at how the world has changed.

> When the first chips came back from the foundry in May, the team pointed its internal AI models at designing software to run benchmarks such as SemiAnalysis’s InferenceX. On DeepSeek’s multi-head latent attention kernel benchmark, performance climbed from 0.31 percent of the theoretical ceiling (set by the chip’s compute and memory bandwidth) to 88.94 percent in roughly 40 hours. Ho says this result is repeatable, so the time between when foundries deliver the first chips and when production ramps up can be reduced. “All our schedule assumptions are going to be based on the fact we have this capability now,” he says.

How I write with an LLM: for each page “suggest up to 10 word changes to increase clarity.”

Is there a downloadable technical report somewhere?

This person knew they did not prove the Collatz conjecture and others independently figured it out within hours. Not sure this is at all relevant, other than pointing out how trivial it is for the community to understand errors in lean4.

I agree with search, which is what mathematicians also use over longer periods of time. But it is not brute force search (and neither is alphago’s search or modern stockfish, though both still search at depth and speed higher than typical human).

Not sure what you mean. Here is what happened in that case: https://news.ycombinator.com/item?id=49137060#49140177

You misunderstood. This new project has 2,900 LOC. Maybe the spelling change is too subtle.

Perhaps you did not understand the Fermat theorem proof announcement/repo or the link. The 13 million lines did not use any external, possibly not honest libraries, as the proof eventually only used the fundamental axioms. So for the Fermat theorem formalization, no open open questions remain.

I had good luck with Kevin Lin’s tip for Astra: “Can you radically simplify the implementation?”

You jest and that is OK. Brute force search is not something you can do over math problems of that difficulty or anything with combinatorial complexity.

To me it feels closer to taking the top 10k human mathematicians on a large retreat for a year and having them self organize to collectively solve this problem—not kids and easter eggs.

As far as I understand the 10k agents worked on the proof. The lean formalization came later and was easier/faster than getting the proof.

If you work with distributed systems, you still call that scenario a success. On the other hand, if the 3/4 of agents going off the rails bring down the whole mission, that is a failure. The latter would have been my guess with current models scaling to 10k agents.

I am not an expert in lean4, but I could follow parts of the high level lean definitions of the problem statement in the repo. A lean bug would be a fun scenario; I am certain this proof will receive the deserved scrutiny, and if it uncovers a bug, it will make the story even more exciting. It is extremely unlikely to be the case, however, because the 10k agents working on the proof didnt use lean, so it would have to be a math logic error that translates to a lean bug—perhaps something the agents picked up during training?

I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.

Other than the undeniable breakthrough in math, the important point is the ability to orchestrate 10k agents to productively work on a single problem, which creates options:

> OpenAI, meanwhile, says its experience with Navier-Stokes could open the door to solving puzzles with more practical relevance. “We are now able to spend millions of dollars on a problem that we really care about and that really matters: developing new materials, finding cures to diseases,” Bubeck said. “All of those things that we have been talking about for a long time—now they seem to be at our fingertips.”

They managed to solve a problem that was beyond current human ability.