isomorphic_duck

You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).

I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.

An idea to start could be something along the lines of an agent given tasks incrementally, where the agent would be required to extend/generalize a piece of functionality. A possible reward could then be the amount of code the agent had to add/remove to complete the task.

Not all points being collinear does NOT mean that all 3-tuples of points are non-collinear! The hypothesis of the theorem is the former. And what it proves is that there is at least one such 3-tuple.

A key point I feel that’s missing from the discourse is that alignment/safety is basically an impossible problem as of now. Even the “guardrails” that these closed-weights frontier labs set in are laughably primitive which can provably be broken through.

The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.

how does biology depend on "dogma and mysticism"? I am really curious - a Google search yielded nothing much relevant.

Why did you make a new account to spam AI comments?

I didn’t think I would see this take on HN of all places. You can’t appreciate Higher Topos Theory before spending a decade’s worth of effort in pure math - does that make much of Modern Algebraic Geometry “garbage”?

It might be more helpful to think of it as small changes in “genome-vector” taking place across generations of the species with the filter of it being not bad enough to cause extinction of the bespoke variation, as opposed to evolving towards more fitness as an optimisation objective. When thought of like this, one can imagine how high-dimensional the “feature”-space becomes, which leads to such intricate engineering we see in nature.

> I've had Claude fuck over clean well documented code-bases for no reason

How exactly do you define "fucking over", and why do you suspect this "fucking" was done as a result of a faulty trigger as opposed to the inability of LLMs to write maintainable, extensible code?

"Never attribute to malice what can adequately be explained by stupidity."

I am not an astrophysicist, but I have read that the thick layer of ice is supposed to protect the miles-deep ocean against radiation, which might harbour life.

Tangential, but really looking forward to what Europa Clipper[0] finds in its flybys.

The delay in communication makes ambitious manoeuvres challenging - perhaps advances in AI (and by extension robotics) helps build much more autonomous space rovers. This could enable us, for example, to evaluate the samples by sending wet microscopes with the rover itself.

[0]: https://science.nasa.gov/mission/europa-clipper/

We operate and think about subjects like Higher Topos Theory, Information Geometry and Algebraic Topology, which are several layers of abstractions removed from anything that can be termed as a skill “specialised to our survival”.

You have to understand that the median human is terrible at (almost) everything. Humans, the only examples of general intelligence we know, are economically valuable precisely because they can train themselves to specialise at a (relatively) narrow task over time. You don’t measure how good a coding model is by how well it programs relative to Doctors, or how well it can prove theorems relative to baristas, or how well it can write coherent novels relative to programmers. That would be a dumb metric.

If Claude Mythos and Fable 5 are the same underlying models just with different safeguards, I fail to see how TerminalBench has them at different scores.

Continual Learning? Why is this even a question? Isn’t it a well-known glaring issue with the current models? They cannot learn/adapt to new skills (in any permanent sense) once they are deployed.

No, the purpose was to create a (automated) test set in the first place. The author builds an LLM judge which can score the LLMs participating during test-time. That would be why the author used the strongest model (Opus 4,7 at the time) as the judge.

My biggest gripe with the discourse around AI, especially by programmers with hubris about Machine Learning, is the idea that LLMs can’t come up with “novel solutions”. They can, and they have. CoT[0] is how LLMs can output tokens in “reasoning space” to guide their “thinking” to produce absolutely novel solutions. You can imagine reasoning being multi-layered, where the top layer is an abstract heuristic (examples of which can be “try special cases”, “try solving a part of the problem with relaxed constraints”). The lower layers become more and more concrete with the details of the problem, and the result is a solution of the problem.

You don’t even have to understand how modern reasoning LLMs work to be able to tell that your perception is warped and doesn’t reflect reality - there’s plenty of news to the contrary - OpenAI resolving a major Erdos problem[1], the First Proof endeavour[2], amongst others [3].

[0]: https://arxiv.org/abs/2201.11903 [1]: https://openai.com/index/model-disproves-discrete-geometry-c... [2]: https://1stproof.org/assets/docs/report.pdf [3]: https://archive.ph/2w4fi

Also 100, which speaks to the infinite demand of software - We will never run out of things to program as long as there is a single program around.

I can’t imagining investing into these frontier labs for the simple reason that Open Source is very likely to catch up in a relatively short period of time. I don’t see how OpenAI/Anthropic could then continue to serve their models with such large inference margins.

I would say continual learning is the big missing piece that someone even from 2002 would realise.

It remains to be seen if LLMs would do any good in the "theory-building" heavy fields of math. They have certainly proven themselves in branches of math where the progress is verifiable, but fields like AG commonly have papers that do not concretely solve a problem but provide a new perspective/framework. This is iterated upon if other mathematicians find the construction rich and interesting enough, which eventually leads to breakthroughs.

LLMs have yet to show that they can meaningfully make such helpful abstractions. Not saying that it can't be done, but I wouldn't write such doomer posts just as yet.