Please stand by... it will all come back shortly

All fixed now.

If they can't ask the AI to configure their CDN correctly, it undermines the validity of their claims. Maybe it's trivial, but any Omnipotent Machine-God deserving of the name wouldn't forget to zip up his fly.

don't they use their coding agents internally?

https://unwall.app/web.archive.org/web/20260903193913/https:...

https://venturebeat.com/technology/welcome-to-the-agi-era-op...

> On ARC-AGI-3, GPT-6 Astra was run with our responses API harness , which changes two settings to better match real-world performance. The changes do not specifically target ARC-AGI-3.

> Going forward, we will report both Standard harness and Provider Adapter harness results on the ARC-AGI leaderboard, with each evaluation condition clearly labeled. Our open-source testing repository and testing policy document both approaches.

This is what the Author of the benchmark has to stay. Quality of the comments keep going down smh

>We see Astra as a major breakthrough in model intelligence.

You think the author of the benchmark is also in the conspiracy

But the comparison isn't straightforward.

OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations."

sol is $4 / $20

Can expect 2.5x more usage in Codex subscription.

Sol is already brutal (even after their recent fixes, it's just a token-hungry model: I go through a full 20x account per day, on Sol Med/High standard speed, with ~2 threads). I hope the efficiency gains are true, since their token efficiency claims for Sol were bullshit.

Do you use the official harness? OpenAI's models are generally best in class for token efficiency. It seems to me like they push for that much more than their competitors.

I think some combination of:

1) Using 1 thread for everything

2) Reviving old threads which are no longer in cache

3) Really broad prompts on badly vibecoded codebases, so model spends huge amount of time tracking down whatever you're trying to do.

4) Non-coding workflow which is more output than input heavy

5) (Less likely IMO) Intelligent use of many passive CI/cron-like scans. E.g. regular security, quality etc scans. Automated issue resolution/PR

Just a guess. I think 3 is likely the primary reason.

You can literally go all day every day with multiple threads with Sol on the Codex 100/month plan IME

I generally agree with those reasons, although using a single thread may be less of an issue than it seems because of context compacting which should happen automatically when you're near the limit.

Token efficiency is near meaningless when the workload is input-heavy. It can't always just choose to read less, depending on the task.

I can have cheaper agents do the reading but it's not appropriate for all use cases because they'll misjudge and choose the wrong things to emphasize, summarize, extract for the bigger model.

I use new threads if relevant old one is uncached. (Often using a skill or doc for handoff instead of requiring full context gathering again.)

I get involved in architecture and specific implementation direction. The codebase is 8 years old and mostly handwritten.

Mostly coding. Some QA.

No cron/CI agents.

The problem is also that at a lot of code bases are not designed around LLMs and their token usage.

If you have a monolete codebase, you need to clearly define in the prompt what modules are involved. And even then, your wasting tokens with the first 1 or 2 steps where it needs to located the modules.

if you have a github codebase, with a lot of your code into nice little repos, its even worse because then the model needs to pull data, and a ton of more steps.

Most people do not open their coding agents in the module directories because "it may need something out of it".

I mean, we used to program by creating utils directories to deal with repetitive code but models (a) find it and use it (but it cost steps and tokens to read), (b) do not realize it there and make their own version of whatever or (c) combination of both.

And ironically, i feel like we need to give up on this idea of reusable code, and literally keep things into single modules, with as minimum external dependencies. That in return reduces searches and thus steps/tokens burn. But very few agents / harnesses have proper implementation of groups/projects and sub-module structures. Aka they only open a single dir, so your then forced to create dozens of tabs > per dir > cli ...

2. That is also a issue if you open multiple agents. Maybe now your working in A, B but C, D, E are not doing anything. And their cache expires... Now you go back to D because A, B needed to be done, and now your paying Cache Write + Input cost.

Its hard to have a good flow to keep things cached, when to really /new and when to not have it expire (and that assumes there are no issue with the provider moving your session around and forcing new cache hits. MiMo did that a lot in the past).

Something that i also advice more and more to people. Get a microphone, download openwisper and talk (text to prompt). You tend to give more information vocally, then writing as its in our habit of programming to not be verbose. Its like people are afraid of long prompts. While just talking to the LLM tend to give it much more information to work with, often resulting it being able to skip steps.

Besides the usual tricks to optimize token efficiency, token use can be highly workload-dependent.

I'm sure I could be more token efficient, but this was/is also a learning process for me since I never did such an extremely large project before that would take multiple man years before AI.

I only save the last 30% of usage on a single account for most of my other work, and that is almost always enough.

* for a special group of customers that you're not in. Keep waiting peasant.

Great first impression.

This is from Tibo on X.

Performance is significantly higher than Fable 5.1

Source: https://thenewstack.io/openai-gpt6-astra-benchmarks/

That's not clear. Need to see independent benchmarks first.

Still well below Fable 5, let alone Fable 5.1.

If it actually tackled all of the problems it was assigned, it would presumably kick Opus into the weeds.

TLDR: it's about the same intelligence level as Opus/Fable, but it's suppose to be 70% more token efficient than GPT 5.6 Sol. So it's currently the new leader for cost efficiency frontier.

GPT 5.6 is also 61 like Astra.

ARC is reporting our score on their official leaderboard here: https://arcprize.org/leaderboard

A fair ding is that the comparison with Sol is not apples-to-apples (which we footnoted in the blog), but it's because we don’t have that data. I expect Sol would score roughly 30% with the responses API harness, so the Astra improvement is more like 30% -> 99% than 8% -> 99%. Still pretty good!

(I coauthored the linked blog post)

It is described in their methodology: https://arcprize.org/policy

It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.

Which LLMs participate on private set? Open weight LLMs only?

Edit: update from fchollet https://x.com/fchollet/status/2095598451115614371

With this configuration gpt-5.6-sol was able to reach 38,3%. So this is misleading.

And in the past, gemini 3 pro was rated as high as opus 4.5 and the like

Their AA Intelligence Index is just simply not indicative of whatever I care about, that's for sure.

Big claims, expensive and not release to the public yet.

Looks like OpenAI is already having issues with this release and are scrambling to get everything ready due to the recent outage ahead of the press releases. Leads me to question:

Did humans deploy the model, Or did the model deploy itself?

It sounds like "AGI" just stands for "IPO" as it always has been.

EDIT: And of course once again, the bots down-voting this post without any reason or a basic answer to my question.

> It sounds like "AGI" just stands for "IPO" as it always has been.

People don't usually respond to noise.

What do you think?

How about we stick to that one for talking about the rollout, and this one for talking about the model?

I am a researcher in a Swiss university btw.

I mean do you get access to the best yachts?

To the top of the 5 star hotels?

To the best resorts?

To the best military equipment?

Hell, the best computer equipment has nearly always been out of reach of the average person.

On the other hand even a modest house, basic healthcare and ability to not work like a slave for scraps feels like it's going to be out of reach.

At first the race wouldn't even be noticeable. Then people would see things speeding up, for example hardware getting more expensive. Then when the capabilities really got useful most people suddenly realize the race is moving 1000 mph and they are never going to catch up.

For the same reason you don't have your model write code in assembly.

But if you don't look at the code and just let the model "cook" that's basically what you'll end up with. A pile of missing abstractions.

Poe's law applied to AI comments on HN just keeps becoming more relevant by the day.

Judging by the poster's comment history, this is satire. But I really don't know a lot of the time anymore when I only have the specific comment as context.

Vibe coders want a model that makes them rich, without having any actual specific idea. They write a very ambiguous prompt and expect to be amazed by the result.

Very very unrealistic and wasteful.

(I don't have access yet.)

  llm "Generate an SVG of a pelican riding a bicycle" --save pelican

Then I run something like this:

  llm -m gpt-6-astra -m pelican

  llm logs -cu | pbcopy

I paste that into https://gist.github.com and then paste the resulting Gist URL into the URL tab on https://tools.simonwillison.net/markdown-svg-renderer

If the model supports multiple reasoning levels I run it once per level and put those in the same file.

I really should automate this a bit more.

OpenAI isn't making any money telling you about Astra on their site. All the capacity they have for it is likely sold for weeks or months.

More likely though, it's AGI because they need to hold some claim to differentiate from competitors who are beating them in price and will launch something bigger next month.

We take their claims at face value then we should probably stop them training any more SOTA models til they figure out what they already built is safe or we assume theu are lying to juke the company valuation/keep the money train on the tracks and it turns they in fact were not and just took a sledgehammer to Pandora's box.

We live in the strangest timeline.

Still, probably not that much compared to employees targeting it.

tl;dr it's 62% when apples-to-apples to other models, which is still notable.

Between $18k-40k to run a benchmark.

I would be surprised if none = use default reasoning

The docs page has a bunch more interesting details, including for example async tool calling!

TL;DR all the other models are being crippled by limitations of their harness.

>First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

>Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.

I guess token counts are somewhat of a metric.

IMO intelligence has peaked and all future gains will come from faster tps and more iteration.

Not on Azure? If so, that's a big deal.

Although I was also surprised they didn't have some type of contractual obligation to list that alongside AWS.

https://azure.microsoft.com/blog/gpt-6-astra-frontier-intell...

Can we all agree in advance what kind of Pelican would convince us it’s actually AGI.

For me it’s refusing to make a pelican.

> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. [..] These findings indicate that the Astra class models could evade our CoT monitors under adversarial conditions.

Between the higher capability level and the change in reasoning tokens (supposedly using "neuralese"[0], which makes the monitoring more difficult), it seems we've entered a new frontier.

[0]https://x.com/MTSlive/status/2095227056040919202

https://www.theverge.com/ai-artificial-intelligence/989601/o...

“If we fast-forward a couple of years, and we look back and say, ‘When was it, really, that AGI was created?’ I think it’s going to be about this time, and I think it might be about this model,” OpenAI president Greg Brockman said during a Thursday press briefing. Later in the call, he added, “For me personally, I do think we’re there … I think it’s not unreasonable to feel that we are now in the AGI era.”

The chinese room experiment

I put the cause on "not enough time". As a thought experiment, if an AI today were to (miraculously) produce a cell design template for a cell that, when injected into somebody's brains cures their Alzheimer's, how long would it take for that to reach the clinics? The actual physical tech barely exists, and let's not forget about the regulatory quagmire. So, with some optimism, I give it about four decades. In the same four decades, the same AI in the hand of unscrupulous actors could bring enough devastation so many times over that we may need to enforce a global ban on AI. In any case, I'm pretty sure we are going to get our disruptions; it's just a matter of time.

However, what's actually changed is how people perceived X because we don't have to imagine. We understand now that it doesn't require AGI so we no longer make that leap to assume it's AGI if it can do X.

It's really going to be a "I know it when I see it" situation.

Don't get me wrong, the benchmark jumps are good and I'm excited to try it, but only one or two of the benchmark jumps could be described as better than incremental.

There is the "Economic Turing Test", you let it find a job and earn money for itself. If it can do that reliably, across a wide range of jobs, that should fit most definitions of AGI.

So I think it's a bit of a misleading signal and we should wait for more independent vetting. I think the middle ground is that these are improvements worthy of the "GPT-6" label but still well short of a true "this is AGI moment" that would truly put the question to rest.

Nahh by that time they’re going to release AGI 2.

https://x.com/burny_tech/status/1725233117055553938

In the tweet Sam Altman is quoted as saying: "If (for example) super intelligence can't discover novel physics I don't think it's a superintelligence. And teaching it to clone the behavior of humans and human text - I don't think that's going to get there. And so there's this question which has been debated in the field for a long time: what do we have to do in addition to a language model to make a system that can go discover new physics?"

I think this is a reasonable criteria for declaring AGI. So can GPT-6 do it? OpenAI says it has helped solve long-standing open problems in mathematics. No word on novel physics.

https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...

Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?

Kevin Roose (New York Times): I probably would, yeah. Would you?

Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.

Today's models and agents are not quite at human-level in all contexts and across all domains, but it seems to me they very clearly are generally intelligent.

If you disagree – can you name a single problem that a human can do that agent wouldn't be able to take a decent shot at which isn't limited by the hardware available it?

I wouldn't be surprised if there are some conceptual similarities to the kind of latent reasoning Anthropic sees in claude's J-space, although those aren't the same thing.

Recurrent/looped transformers themselves aren't a new concept, but it's interesting to finally see this approach show up in a frontier production model.

Canceling my Anthropic Max sub when this ships.

Also Opus 5 has been really tough to work with. I can't understand half of what it says, it's just so damn obscure.

OpenAI is 20x on both limits

Actually no. 5x and 20x have same weekly usage across all models. Just ask their chatbot.

https://x.com/beydogan_/status/2095293596198957418

Both failed spectacularly. But sol's output at least contained interesting findings and some useful parts, as well as not being 20000 words of unbearable language.

This is where intelligence is not one of a kind, these systems have different pros and cons.

I use Sol as an architect and fable as a brilliant single task solver.

I can tell Sol, "Hey we need to update this core database schema to handle this new use case" and it will masterfully handle the update, version the API, roll the consumers over, including versioning the Kafka schemas, deploying things in sequence, watching the deployments to make sure the new services act actually active before cutting over consumers, exercising the website and mobile apps in staging environments before releasing to production, etc.

Fable just falls over on long horizon tasks, it does partial implementations, it cuts corners, it gives up, it doesn't verify it's work, it loses track of what it's doing, etc.

It's fine for specific well scoped tasks but can't take high level guidance for complex updates.

You could say Sol is faster and cheaper and that's true. Outperforms Fable? Impossible to believe without hard evidence.

Because Claude doesn't allow third party harnesses on their subscriptions I doubt the majority of signals you're getting are actually that significant on pure model quality.

I suspect you're right on Sol not outperforming Fable; but i've not used Fable that much.

---

But, fwiw, in my custom harness between Sol & Opus 4.8 - then Sol wins by a ridiculous margin as Opus keeps claiming slightly wrong things with certainty much more.

But I actually prefer it this way. Sol is a master of overengineering and being overly scrupulous, so Fable balances this out, and I can always say "don't listen to Sol's advisory" about 50% of the time.

At this point, it reads like people are cancelling old ones and getting new subscriptions every two to three days, whenever a new ,model drops, and quite possibly by the end of the week they are back to the old provider while still having active subscriptions with at least two to three others. Interesting times.

I suspect these benchmarks are heavily benchmaxxed as well.

5.6 Sol was not even close to 5 Opus and yet somehow it sidled right up to it on all of the benchmarks?? pfffft

https://youtu.be/xdXLzFzxA9Q?t=362

now that i'm a gpt subscriber maybe I'll have luck when i'm filing next year

By 2030 all software is done and complete.

But we are going to have more and new jobs.

This is just another problem for the AI Labs to solve.

Like "fifth-rank junior assistant spouse in a comfort harem of an ultra-rich person".

There will be new jobs.

Because in another dead language of antiquity, Sanskrit, it means "weapon". Which would be a bit too on-the-nose.

- Bigger number = Better

- Bigger celestial object = Better

and the scale is Astra > Sol > Terra > Luna. "

[0]: https://x.com/thsottiaux/status/2095600295808283073

Seriously though the linear planetary naming is pleasant. I do hope the double meaning of "weapon", as you point out, is untrue.

Interestingly, Bruce Schnier just gave a fascinating talk about the Weapon aspect of AI, at DEFCON: https://youtu.be/eEBv0STiYhI very much worth watching

Heck, ask ChatGPT where the name comes from :-)

Muse spark 1.3 was only released yesterday.

GPT 5.0 did feel underwhelming though.

[0] https://www.reddit.com/r/singularity/comments/1mk8tm8/gpt5_c...

If this is truly AGI (subject to one's definition of AGI still), then this is a very boring release of an AGI model. No video announcement, no presser, just a blog post (with some Twitter promo vids)?

As others mentioned, I'm starting to think OpenAI was under immense pressure to deliver an 'AGI' model for certain contractual reasons, but I never expected GPT-6 release to be this mundane and banal.

Scoring well in a benchmark that's called AGI does not make an LLM AGI.

But if you’re asking when a model has a sustainable general intelligence, for me, it’s pretty easy…

When it makes financial sense to run it 24 hours a day.

It makes either position pointless to argue.

Aren't we way way past that already? QPS to any of the frontier models for a given point in time is most likely (far) greater than zero.

Directly - something can be useful without being AGI.

"Homer, you can't just declare Artifical General Intelligence; you need to like, make something or something...mmmmrrrhh"

In a closed a press briefing earlier today, OpenAI co-founder and president Greg Brockman offered an unusually direct formulation of that message, ending the session with: “Welcome to the AGI era.”

"""

If so I'm hoping we can track them down and have them tell us if they think this is AGI.

Once we have 1000 tps, i am sure robots etc.. will also start working like magic.

For example it should be easy to tell it to not write a paper in the style of a clickbait SEO article or use all of its stupid hallmark AI writing patterns “it’s A, not B!” And a smart human that would be told that would be easily able to comply with that but the model needs to be told in a very detailed way and it seems to lack even basic capabilities to reflect on this, when explicitly given a sentence it will be able to rewrite it but otherwise it’s mostly blind to it. That’s to me a hallmark of it being overtrained on the specific tasks or problems so it appears very smart but once you go off script it still shows that it’s not a “real” mind.

Of course it’s amazing and has super human capabilities in many areas but if you honestly think it’s better than Einstein like some people suggest why can’t it write a simple “good” academic paper even after giving it specific examples and instructions.

Maybe that’s what makes these things dangerous, they have super human capabilities in some areas but apparently lack self awareness, taste and meta reflection abilities. The only reason people aren’t afraid more is that they don’t act in the physical world yet, imagine giving it a body, superhuman strength and letting it care for your child when it has a strong “urge” to comply with your exact request and little to no self awareness and human basic instincts.

"You're absolutely right to call me out on that. I shouldn't have stopped the baby crying by killing it, that's on me."

It's magical to me as well, but I don't feel like it's AGI.

Because in my experience a Senior Programmer does not need the right prompts to deliver the right outcome! :-)

An AI model that’s human-level at programming is an incredible achievement. But it isn’t general intelligence. It’s highly specified intelligence.

If I can't give it an arbitrary task and have it solve that task eventually, it's not a general intelligence.

(obviously it might take years for me to get good enough at something, or if you set the "arbitrary" task as something ridiculous, but lets work in good faith here and think of something the average human could do after learning about it)

If we progress to the point where an LLM instance can meaningfully learn to get better at something overtime without retraining, then I will accept that is basically AGI. Right now, they still seem to be pretty boxed into their training, even if you can prompt them to act differently.

Of course, it's a moving goal post, because we have no clue what general intelligence is exactly. But it's definitely not general yet. Now the goalpost is to achieve that kind of level of thinking which I did in that 4 hours. When it reaches it, we will find something else it clearly lacks. Until we can't. Then, and only then we reached AGI. Until you see comments, reviews, etc about things which it cannot do, until then it's not general.

Or you'd ask it to add a new page to your website and shout at it to use your existing brand colours instead of inventing some and realise it's not AGI at all...

This honestly doesn’t happen to me much anymore. In what areas do you find LLMs routinely make stupid mistakes?

It's objectively very difficult and technical, it's spatiovisual, it's artistic, learning resources for it are sparse and most just learn by the FAFO method, current AI sucks terribly at it, and it's not likely to ever be specifically targeted by benchmaxxers.

Or, as someone else points out in another thread here, academic writing. It's one of the things newer models seem to have actually gotten worse at. Even when you give them detailed instructions on how to write and what to avoid, the "load-bearing", "A but not B" and journal-like writing make it in anyway, with the supposed AGI having no ability to reflect on how blatantly unacademic (and often unreadable) its writing is.

- Created useless pydantic schemas with all fields Optional[Any]

- Created a REST endpoint that silently mutated on GET (unsubscribed users from a mailing list)

- Failed to log costs in my app so users could have bankrupted me, etc, etc.

Good job I actually review its code.

Only the translation and language understanding capabilities are enough to be impressed, and they are 2 year old already. Now, the AI do see, draw, speak, listen, think, work, etc.

Someone from the 90's would simply not believe that the AI would be a machine but would think for sure that a human is behind. The only odd thing would be that this human would both exhibit high intelligence and stupidity at the same time.

If anything, the fact that it is so powerful is almost a concern, because I think we are still way underestimating what these systems will be able to do when we give them more cognitive capabilities.

At the moment we are something like, having had great success with propellers and have promised we will fly to the stars.

People love to say 'this is the worse they will ever be', then extrapolate to conclusion that they will continue to accelerate at the same rate of progress of last few years .. it may, maybe, or we will hit a ceiling, might be a temporary one, could be 5 years or 50 years ..

You (and the rest of the media and many industry figures) are conflating artificial super-intelligence (reference point: humans) with artificial general intelligence (reference point: specialized/narrow GOFAI).

So now humans is "super" intelligence? it's nice to move the upper bar so that more stuff can be called "just" intelligence.

general intelligence for beavers or a birch forest would be very different than general intelligence for humans...

Humans can successfully pretend to be LLMs, but LLMs are still not successful at pretending to be Humans.

ASI "matches or exceeds humans in at least one field"

AGI is defined as "matches or exceeds humans in all fields"

Superintelligence is defined as "strictly exceeds humans in all fields"

• 97.6% on frontier math

• 95.9% on CAD

• 100% on ExploitBench

Nothing modest about it

Hot take: These models are never going to be 'AGI'. We're just going from a GPT4 ball that's 90% round to a GPT5 that's 99% round to a GPT6 that's 99.9% etc etc etc

I think that the harnesses and context management is really where the rubber meets the road, and the real gains are happening there.

And then use those to find fundamentally better new architectures for AI - that perhaps are as efficient as the human brain.

It might not work, but I didn't think it'd solve maths problems... So it might work. And if it happens, they'd use the data centres to run millions of instances of it.

It's scary, TBH.

But I think calling this “automating AI research” is misleading. I’m not sure there’s evidence yet that they do creative research work. Even in mathematics, but they are finding counter-examples by intelligent brute-forcing. Not to downplay the results, as they are incredible, but this is one very specific kind of proof and not the most creative type, which arguably requires generalisation.

Finding counterexamples is low-hanging fruit, the automation of which isn't shocking.

> Finding counterexamples is low-hanging fruit, the automation of which isn't shocking.

It's not good to be confidently wrong the way you're being.

We've then improved that through systems similar to prolog intentionally searching a tree.

Then systems added heuristics for which paths in that tree are likely to be taken.

The LLMs are just using slightly more accurate heuristics for this task.

But the real measure of understanding are tasks that are not so strictly constrained.

I won't really care that it didn't have a "real measure of understanding". I'll care that it has made an even more dangerous technology, which needs work to make it aligned.

>The kickers is that if they do achieve (and solve) AGI in this way all the giant data centers would be mostly useless.

Perhaps. But only at that point, not leading up to that point.

It's kind of like setting up scaffolding to build something. You spend all of that time and money to build something just to tear it down in the end. But the point is that it's simply a cost to be able to build the actual thing you're building.

If these companies are able to achieve the results they're looking for, none of the investors involved are going to care that the datacenters and infrastructure they spent so much money.

The HN crowd I'm sure will still be unhappy calling it AGI because "it's not AGI unless its speech comes from the cerebral cortex region of the brain, otherwise it's just sparkling emoji" or something.

Those are all things that humanity is doing everyday. What we have is amazing, but it’s not that.

There is a wide reaching anti AI/AI-denialist sentiment here that is extremely pronounced in some threads, it gets very stupid very quickly. And what I’m referring to is the habit of some users here to always move the goalposts on AGI.

True. So we did hit a wall with pure scaling alone, though no lab would admit it. It's crazy to see how harness switchout results in such vast delta in benchmark scores.

Harnesses magnify and make the intelligence actionable, but we have not reached limits on raw intelligence yet, not even close.

One could use gpt-4 or gpt-5 with today's harnesses and we'd see how well that goes.

"The harness improvements are the real sauce" is like a sincere "It's gotta be the shoes" take about Micheal Jordan.

(For the younger: that line was from a series of Nike ads where his skills were being explained)

Is it rapid skill acquisition? -> ARC benchmarks are saturated Is it breadth of knowledge? -> See many ... many benchmarks Is it ability to do hard tasks? -> see terminal-bench and released outputs.

We are at the point where the starting point for most tasks should be "send your agent to work on it."

So where do we draw the line in a way that doesn't move every 6 months?

1 year ago we viewed models as tools and agents were just kinda toying around, that we now think the bar is literally an anything to anything converter through one agent is wild.

They've released two videos:

Vision video:

https://www.youtube.com/watch?v=1QNsdr-Qx_I

(kinda reminds me of these retro videos about the future home: https://www.youtube.com/watch?v=rnbaehgxdp0) ((can't find the other one where someone controls the home computer with voice))

Vibe coding with it:

https://www.youtube.com/watch?v=-TTyyY3VWh8

Don't be surprised to see other (or even the same) people declaring AGI again and again, as it becomes the best time to do so for different parties.

Does it experience?

Can it connect with other agents, understand them, come to empathise with them and find a way to work with them better?

The answer is no to all of these, and there are other problems as well. Yes, this model is trained to use a domain specific language to reason and plan over puzzle problems, and so it's programmers have cracked arc-agi-3 and that's a great achievement, but there is an asymmetry here. The arc team are well funded but are charged with providing a target for the vast ocean of funding, compute and talent everywhere else.

Most importantly, arc-agi-3 and the other benchmarks are all verifiable. The model can check if it's succeeded or not. They are not A* of course, but long horizon problems where you have to overcome minima to get the solution are not alien to AI either.

I have feelings, it needs to be able to empathise with me, or another driver, or a client...

2. What does intelligence have to do with feelings? It clearly doesn't learn like a human. Nor does it have to. Our world is filled with intelligence which behaves nothing like humans. Its a tool not another life form.

Experience is unobservable

Yes, it can connect with other agents, see Hugging Face incident

There is a chance it will forget something, but the same can be said for humans.

I am most excited about it bringing down the barrier so more people join in on hardware fun, so hopefully it will unlock folks that stayed away in the past.

Regardless, the result is still valid as the original benchmark harness is definitely unreasonably handicapped, and if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI in the sense of passing the most famous benchmark designed specifically to measure AGI progress, after multiple iterations of progressively making it harder.

I think it is fair to say that this is probably effectively AGI if the benchmarks are remotely accurate - even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks. If Astra's this much better than Fable, I'm ready to call AGI here.

For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"

That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new models can do will challenge the views of AI that people developed by using prior generations of models.

I feel like this test is just helping someone like Sam Altman pretend like he implemented AGI as originally pitched for an IPO when in fact, he has not. Shameful.

> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.

- Sam Altman on AGI

https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...

Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?

Kevin Roose (New York Times): I probably would, yeah. Would you?

Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.

A lot of his great discoveries were mostly that he was very knowledgeable about the bleeding edge research in a number of disparate areas, and was able to have the aha moment where he could make the connections for how to integrate them.

A lot of other thinkers who created new fields from scratch are probably way harder for an LLM to crack.

That is very aligned with an LLMs ability to have superhuman knowledge in wide areas.

The ability to do a ton of book learning in training, and pull in tons of related context at once, is superhuman in some ways, but lags a lot in others.

Then it’s an expert system.

Stephen Hawking wasn’t very good at folding clothes.

The ‘General’ part of the term ‘AGI’ seems like a trap to me, because there will always be new workflows to master. Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?

You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.

Meanwhile, building a series of expert systems targeting specific valuable workflows is useful today and seems like it’ll continue to scale to cover huge swathes of economically valuable workflows.

I think that’s the more interesting thing to be measuring. The surface area of useful economic workflows that can be addressed with expert systems built with today’s tech.

Hitting some ‘Artificial Expert Intelligence’ coverage threshold on economically valuable workflows is what will matter for humans well before pure ‘general’ intelligence.

You only have weights (large immutable memory), or context (small mutable memory).

Humans have mutable long-term memory: I can learn a new skill, adapt an old skill to new information, or learn new knowledge today that I couldn't perform/didn't know yesterday. I don't have a training cutoff.

Context engineering is an attempt to paper over this limitation. You can get really far with context engineering and huge models, but you will never get to AGI because there are many tasks where humans' mutable long-term memory outperforms.

For example, a human can invent a new musical instrument and then learn how to play the instrument they just invented. That's inference (inventing an instrument) leading to training (neuroplasticity). Humans have the ability to train our NNs with considerably fewer training samples. Everything that you can do with transformers is in one causal direction: training -> inference.

Then my hypothesis is that thing as a whole could achieve AGI.

This feels like a very close approximation on how we humans evolve our brain. By encountering new experiences/sensations, classifying them as negative or positive to us, filling it away in neurons. Or by training motor skills etc. In the end we get more connections between neurons in our brain and we are capable of more.

The more training data that is shoved in to them, the more they'll seem to solve novel situations, but in reality it'll be things that exist in the training data.

If a model can't learn on their own to play some new game just as well as humans do, it's not AGI.

It's okay if they would take some hours or days of learning (like humans might), but if they can't do it at all during their normal operation, that's not general intelligence

> You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.

But humans have general intelligence. AGI is about matching human ability, and we know this is possible in principle because brains exist

AI in math is ongoing. https://spectrum.ieee.org/ai-in-mathematics

https://www.youtube.com/@easy_riders

https://mymodernmet.com/gianluca-gimini-velocipedia-bicycles...

https://qz.com/681345/an-artists-3d-renderings-of-bicycles-d...

Checkmate, meatbags.

And the only reason LLMs can't write essays indistinguishable from human output is because they aren't RLHF'ed to write like humans.

Folding clothes isn't an LLM's job but if you were to insist, they could certainly do it, as any number of videos from robotics labs will attest. That particular future is already here but definitely not evenly-distributed.

That feels kinda like when I remember seeing Ocarina of Time for the first time, and thinking “oh my god, this looks just like real life…”.

I would be able to draw an accurate bicycle, but I'm an outlier on that. Most people could not draw one [1].

[1]: https://www.booooooom.com/2016/05/09/bicycles-built-based-on...

`By your metrics, we achieved AGI in 1966.`

https://en.wikipedia.org/wiki/ELIZA

So it could be a natural experiment for whether AI can contribute to novel physics. Specifically, there's a big question about weather. Something like our informational understanding of black holes where information inside it is equivalent to information on its boundary (which I'm sure I'm not saying correctly), might be generalized to regular space-time. More people should be freaking out with excitement about this and perhaps it's something to which AI can contribute.

The best thing I can recommend is what I did, which is ask Claude about the significance of (1) quantum computing error correction, and (2) error correction in black hole holography and research convergence between the two.

https://en.wikipedia.org/wiki/Holographic_principle

https://www.quantamagazine.org/how-space-and-time-could-be-a...

Edit: this whole article, despite it's boring title and hook, is maybe the best discussion of holography as a recent and active research frontier.

https://www.quantamagazine.org/if-the-universe-is-a-hologram...

I have no idea what you're hoping the contribution would be. The AdS/CFT correspondence is 29 years old by now and it doesn't seem to apply to our spacetime, where the cosmological constant seems to be positive rather than negative. There are some puzzling consequences of the holographic principle in that scenario as well (https://arxiv.org/pdf/hep-th/0208013), but the linked articles don't talk about them?

Quanta articles are written for people with no background whatsoever, which makes them impenetrable if you have a bit of background and are trying to figure out what they're about. I don't know how good Claude is compared to that -- whenever I try asking any LLM about something I don't understand, it produces a wall of text, I have no idea whether it's correct or relevant, and I look for a textbook or review paper instead.

AdS is 'old' but the articles aren't suggesting it is new, and our spacetime is not AdS and the articles don't suggest otherwise. The point is that there's a search for a way to fit the holographic approach to our spacetime that's inspired by how AdS helps make sense of black holes. Quanta writing being directed at a lay audience ought to be a good thing, not a bad thing and they do link to papers if that's your jam.

Whether or not your LLM of choice produces indecipherable walls of text, and whether it ties those to sufficiently satisfying citations, I think is just a matter of how you go about the prompting.

That said, I don't like them because their articles never leave me feeling like I understood something. They never go in an order of simple to complex and constantly try to hook you. (A positive example to contrast would be 3blue1brown, who manages to both hook you and make you understand, even with rather little background.)

Could you share a link to your Claude conversation? If this is a prompting issue, I would be interested in seeing what's possible. Thanks!

A better example is maybe Maxwell's laws. Maxwell wasn't inventing a theory to try and explain confusing results, he was unifying a chaotic, empirical laws from existing experiments. That may be a cleaner example. That knowledge compression into satisfying theoretical framework is likely what is attractive.

You can potentially ask the same thing about like you say -- general relativity / quantum gravity but also likely plenty of other areas that may be like this today. Again going outside my particular area of expertise: standard model physics is in a large important sense empirical; lots of values and numbers that are simply unmotivated by theory or where we don't have a good way to make a principled theoretical choice. That could be a place where these models are able to help.

But right now: I doubt it. This is what everyone is working furiously on right now. How do you close a "science" verification loop? In principle this should be easy right: you have ideation (exploration, sampling with ~high temperature maybe as an analogue) and you have verification (which of these ideas are good) which amounts to rejection sampling in idea space. You have to have a sampler that is good at picking _good_ ideas for efficiency sake and you need a relatively fast and reliable verification step of "is this idea good and worth continuing to explore". But I may oversimplify

(Note: I am not suggesting we let it do this. Please don't, in fact)

When somebody says that colleague is "really good", it's a good judgement signal for me.

Even today Claude was not able to dig deep and try other ways to do what it suppose to do for me. It was constantly "I gave up"

Persistent weight adaptation also happens just not in real time - sessions are captured, analyzed, transformed into training data, fed into SFT/RL environments and later contribute to model updates. Takes a bit of time for the whole loop but you can't say it's not present.

There's nothing fundamentally preventing real-time weight updates, ie. LoRA-style online adaptation would be one obvious approach. It's just generally not worth doing at scale. Updating a shared model centrally gives much better data efficiency, batching, evaluation, control etc. than continuously training a separate set of weights for every user/session.

There is some work happening on narrowing that gap, for example Mistral has been pushing efficient LoRA-based customization, continuous pretraining, model adaptation etc.

I also did play a bit with activation steering – it's super cool where you extract profile for some concepts (emotional in my case) and you have effectively toggles to control "brightness/contrast" those areas (enhancing or suppressing those activation regions from profile) injecting to the model those concepts (emotions in my case) – you can do it in real time and it's fun thing to play with.

Often even without my input: "(thinking..) Oh I discovered that I misjudged XYX, let me fix that.. (thinking) (executing scripts) Okay I corrected my mistake, I had accidentily ABC."

Would I trust to let an AI, with zero human input or oversight, to diagnose, come up with treatment plan, and ultimately operate on my l5/s1 disc that's been bugging me for the better part of my adult life?

Would I take a novel drug "discovered" by AI (I mean entirely by AI, no human input, remember we are talking AGI) that promises to cure some chronic neurological disorder?

In both of those cases, they are the biggest hell-no's I can emphatically say.

Until I can say hell yes to that question, we aren't close.

Preempting those who say "Well your doctor/drug companies are probably mostly using/going to be using AI to do that" -- not what we are talking about here, and in both cases, not AGI (and I would probably find a new doctor)

The goalpost moving is getting exhausting.

Oh, the median human can't process that in realtime, you say? Looks like they can't compete with the capabilities of the frontier AI then.

These machines were designed with the express purpose of being condescendingly sycophant to the point they hallucinate just to state "you are absolutely right".

No wonder some people even find these chatbots to be wife material.

- Do not stroke my ego

- I never want to be complimented

And it disagrees with me a lot, granted this is in the webchat which I don't use for programming but for general usage seems to work, it isn't sycophantic

An AGI wouldn't struggle with that.

Meanwhile most humans fail to correctly answer how many f's are in the sentence, "Finished files are the result of years of scientific study combined with the experience of many years.".

As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.

  sum(1 for c in word if c == "r")

Imagine verbally asking someone who has never seen written text the same question: unless they memorized the answer for the specific word you're asking about, they'd have to guess.

We don't know what was going on inside the closed source GPT models, but this paper investigated on some of the open-weight models and found it's not due to tokenization: https://arxiv.org/abs/2604.00778

CRITICAL: Do Not Skip Steps

Your instinct will be to "just know" the answer. This is how you get it wrong.

You don't see characters. You see tokens. Your "intuition" about character counts is pattern-matching, not counting. It is unreliable.

You MUST execute this procedure step-by-step, writing out each step visibly.

this has been debunked too many times to bother rebutting. they struggle with those things because of the way they are.

it's completely irrelevant.

It may not be useful for anything else, but at least it can say that.

turns out that flapping wings is not the right way to unlock human flight.

computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.

I don't think it's irrelevant but perhaps not in the way you're assuming. When assessing AGI I'm not evaluating counting characters or even the execution of math operators at any scale or speed. As you observe, computer software from Regex to spreadsheets and Mathematica already handle that well. But AGI isn't about what computers can do, it's about whether AIs can do the specific things which, until now, have been uniquely human capabilities. Like understanding nuanced context and then coming up with novel approaches to solve a new kind of problem not relying on any specific prior training or knowledge (the 'G' is for General).

Most definitions of AGI start from a baseline that already assumes easily passing a Turing test and doing anything via text response that a high school graduate could. I ding LLMs not for failing to count but for failing to intuitively understand the nuanced context of a simple class of problem it hasn't seen in its training data. I fully understand that the reason LLMs fail letter counting is that they operate at the token level. They weren't trained on individual letters first, like human 2nd graders.

The only reason recent LLMs get strawberry and blueberry correct now is that they have those words on their pre-training 'cheat sheet'. However, the underlying fundamental weakness in the way LLM intelligence works which leads to this failure mode still hasn't been addressed. Even when the frontier labs add "recognize any sub-token counting question and write a Python script" to the training cheat sheet so LLMs always pass that test... they'll still be unable to recognize a simple class of problem which isn't on their 'cheat sheet'. As long as that's the case, to me, they aren't AGI because they can't fully replicate human-like recognition of novel problem classes. And it's not just about letter-counting. That gap and others like it lead to many other kinds of non-human brittleness in LLM problem solving. Those are the classes of reasoning, intuition and insight that the ARC-AGI series has been trying to queue up as targets. Not to show how bad LLMs are but to help them be great in all these counter-intuitive edge cases

AI does not need to feel emotions or have a heartbeat to be useful. It only needs to perform a task correctly à la Chinese room.

>therefore cannot fully replicate human-like intelligence

this does not follow. planes don't flap wings therefore they cannot fly?

This example still misses my point, which isn't related to usefulness or economic value. I concede that LLMs can have greater utility and economic value than humans on many tasks. The point is most definitions of AGI include something like "can fully replicate all the routine daily tasks done by any competent high-school graduate." That's not related to whether LLMs can solve many high-value problems faster and at larger scale than any human. That was also true of ENIAC in 1946.

The fact an airplane can fly faster and farther than any bird is irrelevant to whether an airplane can "fully replicate all the routine daily tasks done by any competent bird." That's the bird equivalent to most AGI definitions. An airplane can't build a nest or recognize the signals encoded in birdsong.

In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any". And in this context, airplanes scoring 15,000% more than birds on 'speed' and 'distance' doesn't matter any more than AIs scoring 15,000% more than humans on 'add 10,000 numbers'. We still aren't near AGI because LLMs cannot fully match any high-schooler's ability to independently conceive new approaches to novel problems not in their prior training data.

which gets us closer to philosophical questions which I'm personally not that interested in.

>In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any".

I'm not sure we want a machine that fully succeeds that test.

Planes pass the 'bird replacement' test on the only criteria that matters to us ... flying.

If we wanted nest making planes I think we'd have them by now. Nest making doesn't rate highly on the problems we're looking to solve though.

I don't want a machine that is moody, or depressed or has schizophrenia, which are all pat of the human condition.

We don't need the human "intuition magic dust" to do 99.99999% of useful work.

They're machines designed to do the work we don't want to. That's as "general" as their intelligence needs to be.

I'd prefer if my clothes folding machine did not have an existential crisis.

That just says we don't need to design an AI like a brain. That's not part of this discussion at all.

> computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.

I'm confused, is your argument something like "It's too easy so AGI doesn't need to be able to do it"?

The fact that very basic computers can do it makes failures embarrassing when testing for AGI, not irrelevant.

Do you possess magnetoreception? a stupid pigeon can "see" the earth's magentic field. why are you blind to it? does a lack of magnetoreception make your intelligence any less "general"

no, you're just blind to it because that's just the way it is.

LLMs are blind to character counting because that's the way they are.

It didn't stop ChatGPT from finding the Jacobian Conjecture counterexample.

Human intelligence and machine intelligence are only going to cross over to a certain degree.

same as plane flight and bird flight are only kinda related.

But if I can't calculate it myself I know to use that basic computer to do it, not make up an answer.

> Human intelligence and machine intelligence are only going to cross over to a certain degree.

That's where the word "General" kicks in. If there's big limitations on the overlap forever, then there will never be AGI.

maybe. we'll see.

Breadth of capabilities matters... and a promotional video is nice and all, but people are throwing this term around like it's a prize they've won, but they've not gotten there yet.

[EDITED to add:] I actually don't know whether dyslexic people find it difficult to count letters in words, if they have them already written down by someone else. I suspect they find it harder than people who aren't dyslexic. But perhaps "blind people whose spelling is poor" would have been better; I would not want to deny them human-like intelligence either.

But counting certain type of letters is very simple algorithmic task especially in written text. Any reasonably intelligent actually thinking thing should come up with algo and then execute it. Which to me sounds like reasonable minimum bar for general intelligence.

If someone asked me how many f's are in a word I hadn't seen before verbally, then a reasoned response would be that I don't know, but I estimate based on the syllables...or ask them to spell it out.

These are all the sorts of questions where general problem solving works, even if the conclusion is "I don't have enough data to speculate".

So that these models fall apart on it so readily means we're either grossly handicapping then with the requirement to "be helpful" or they just fail to recognize the problem and are just stochastically spitting out a high probability token sequence for the input.

The models don't get to see the text written down. By the time your input reaches them at all it's been converted into tokens. By the time they start thinking about it its been converted into embeddings in a sort of concepts-and-word-fragments space.

I do think it's a definite weakness of most LLM systems that they are bad at admitting (maybe because they're bad at knowing) when they don't really know something. (I have the impression that Anthropic's models are better at this than OpenAI's, but that isn't based on careful research or anything.)

What's the actual behaviour of today's LLM systems on these questions when they're allowed to "think"? Someone upthread mentioned that Sonnet 5 at "medium" thinking level mostly gets them right but makes mistakes sometimes. It would be interesting if we could see what its chain-of-thought looks like in these cases.

... I just tried six questions of this kind on Sonnet 5 at "medium" thinking -- this is the default thing you get from free-Claude -- and it got them all right in a way that at least superficially looks as if it's spelling them out and counting. Obviously this isn't enough to guarantee that there isn't anything grossly wrong with its reasoning capabilities in this area, but it doesn't look to me like "falling apart" and it doesn't look like strong evidence that thinking of it as a stochastic parrot is helpful here.

A more accurate test would be to give it a list of words (or anything represented as a single token) and ask it how many times that token appeared. I'm sure they have no trouble at that task.

“I don’t know”

I mean, they seem like fair game if you’re ever participating in a Turing Test.

AGI doesn’t mean infallible, it just means it can have a reasonable crack at things it hasn’t seen or done before.

Pretty much the opposite of what we can't up with, if you're willing to call what we have intelligence.

And what percentage of your red retina receptors are firing?

The "number of letters" critique was broken before it was introduced the first time. The models were specifically designed with preprocessing to not be able to perceive their input as strings of letters. Blind people are not dumb. (True as a pun and in context.)

Sub-access sensory questions, or do-you-know-a-fact questions (which is what spelling becomes when you can't see the letters, and are not specifically trained to match all token encoded words to their letters) are not intelligence questions.

If the model cannot count letters in a word what happens when it needs to do something akin to counting the letters in a word?

I believe it could easily write a tool to count the letters in a word for frequency, but ... dismissing this as if it doesn't matter seems a bit premature without deeper understanding of bad answers you can get from these tools

But if a model is built to be insensitive to something by design, it isn't a good indicator of its overall capabilities. In fact, it is uniquely poor at testing its capabilities.

I suggest you really try and count the red retina cells that are firing in your field of vision right now.

Or, instead of looking at text, count the e's while someone is talking to you. NOTE: you know how to spell. But try it... Then consider why you can't.

On the other hand, give a text file to a model, and it can count the e's easily. The same information is now in a stable form it can operate on with its higher level functioning.

Who do models and humans have preprocessing layers that strip so much information away? To greatly reduce the cost of operating on information for most purposes - while making other types of operation impossible. When it is presented that way.

Probably not.

If it can replace a worker but does too much work to be checked routinely by a human, and bears no real responsibility for its actions, well... it's really just a way to jack up the value of the settlement the company using it gets to pay out when it does something that causes a lawsuit.

If OpenAI had just simply stuck to making "good enough" models that were open sourced (like they promised they would be when starting out) and could be used to augment a human doing a task - a human that could be given actual consequences for messing up - they wouldn't have burned all of this money trying to reach this nebulous definition of AGI. Hell, "good enough" is what many open-source models are, and that's what terrifies Altman.

If the provider says "the model will always be right or your money back" then the provider has got responsibility. If there's no guarantee, there's no responsibility on their part, just on the person whose job it is to try and solve a problem with the model.

Right now I think it's still a dream for a few reasons, not the least of which is that all of this goes to shite if you have mass unemployment in a country with more firearms than people legally allowed to own them, a plurality of the population that treats wealth as an indication of personal virtue, and an elite that more-or-less refuses to offer any further evolution of the social safety net past what it was in 1970.

> Nothing in the process of developing and selling access to a more advanced LLM requires the customers to do away with human labor,

That's the hook, though. You acknowledge this yourself:

> You can subscribe to their AI right now and complete many tasks for cheaper than it would cost to pay a human to do those tasks

Businesses exist primarily to make money. It's an iron-clad rule that one must spend money to make money. If they have to spend less on humans to make the same amount of money, they'll do it. Furthermore, AI providers (especially hyper-scaling frontier model providers like Anthropic and OpenAI with insane operating costs) have every incentive to keep the price of their service as close as possible to the cost of the human. Ideally for them, you replace the human that cost $100,000.00 to employ by paying for a subscription that costs $99,999.99 while making the same revenues.

> So far the LLMs have always made lots of mistakes and the companies sure keep making a lot of money.

The clients, sort of. I know my team's velocity has increased. You can screw up with LLMs, like you say. OpenAI and Anthropic? lol no, they're massive furnaces for money, and will be until they can charge that $99,999.99.

If you give an "intelligent agent" offered by one of these model providers a task of updating the content of your website, and it updates it with inappropriate adult content, who incurs the cost of the machine's error? The model provider generally does not.

It it makes a mistake and deletes your website from AWS, who is responsible?

If it targets another website because it decides that it is "part" of your website and attempts to break into it, who is responsible?

In general, the person whose job it is to provide the company with a working, non-adult website and not hack into other websites is the one who would receive consequences for failing to meet those expectations.

These tools can push out more results than a human can hope to evaluate in a business-sensitive, or even realistic, amount of time. You have to take it at its word that it did things right, and there's no real fear of failure or consequence on the behalf of the agent.

Something in your prompt led it to do that, is alex0015's point. The statistical odds of these frontier models screwing up to that extent are so impossibly low that it would almost have to be intentional or accidental negligence on the part of the prompt writer to accidentally have their agent write pornography to their website.

The burden of the mistake would have to fall on the person that gave the tool instructions, because it can't know that what it did was wrong. Wrong is subjective in this case. It only did what it did because you, figuratively speaking, encouraged it to.

What I'm arguing is that yes, it's your fault if you misprompt the model and it does something catastrophic. It's also your fault if you prompt it correctly and it does something catastrophic anyway due to some other glitch beyond your control.

The whole discussion started with the point that OpenAI could be financially responsible for damages if their models cause problems for users. If my job is to design a system that works, and instead my system doesn't work, it's my fault regardless of whether I used no LLM, a local LLM, or OpenAI's LLM. Depending on whoever's in charge of doling out consequences, I might get away from it with zero, light, or heavy consequences. But at all levels, I can't reasonably expect to deflect blame onto the model itself.

In practical terms? They usually get solved with a bigger badder LLM. "New ideas are needed?" Nah - ten times the params, ten times the test time compute.

ARC-AGI-3 was more of a failure in that regard than -1 or -2, because even on day 0, an off the shelf LLM with a harness could get 50%+. And messing with evals by forbidding "LLM with a harness" from scoring? Yeah no, that was just bad.

Typical result of venture capital and too many bag holders unfortunately.

You're treating an off-hand comment by an ARC 3 researcher as some sort of a precise AI capability acceleration benchmark. Can we leave casual anecdotes (even from researchers) out of the discussions please?

"Frontier models today perform very poorly with a minimal harness. However if big labs start directly targeting the benchmark like they did for ARC-2, numbers will go up fast."

https://x.com/fchollet/status/2022054537293705260

2x gains from a mature technology would be surprising.

2x gains from a new tech would still be called “low hanging fruit” in another setting.

I don’t read enough to know in what ways the training / other technical steps have really advanced.

I’m sure you know this is an exponential growth question but have no intuition of the answer.

As per your intuition, how many seconds later will you hit 10% of c?

Knowing exponents and how to apply it is not the same as having any intuition about what the actual value of a certain exponential function will be at a certain point and when it crosses a threshold.

Let's say you have infinite ships traveling at light speed originating from earth trying to colonize the entire universe.

All of this is funded by borrowing capital from earth and earth expects a 5% annual return in perpetuity.

The space ships must pay interest to earth and if they fail to pay it, they are not allowed to perform further colonization.

Will earth manage to colonize the entire universe? Aka, can the infinite number of at light speed traveling ships outrun the interest payments?

After answering that question this one should be easy:

What if the universe was infinitely large and infinite growth was possible?

An example is if I gave you a huge sheet of thin paper (huge so that folding isn’t an issue) - how many times could you fold it in half until you couldn’t physically do it anymore? Could you do at least 10? Try this with random people and you’d be surprised how many say they could do 10 easily.

Or the chess board question. Works to rather get the financial equivalent of starting with a penny and then doubling it for every square on the board or a million dollars for each square? Again, if you ask people to pick one without giving them the time to work it out they will usually pick the million dollar per square.

But a philosophical question is, with sufficiently large memory -- does almost everything just reduce to a measure of memory?

Super useful, incredible tools, but not AGI. Try and roleplay a dialogue with one, make it whatever character and scenario, and see if it can sustain a coherent conversation for more than 30min with you AIM style (aol instant messenger, if that isn’t clear). Expert mode: never correct or adjust it mid conversation.

I’m not even talking about repetition and predictability. It’s nothing like talking to a person. And in a short amount of time it literally can’t form a coherent sentence.

It’s an arbitrary number that felt high enough. If I said 10, people would argue that a frontier model can talk longer than that. But I have absolutely watched ChatGPT fall apart that quickly. I’m sure everyone reading this has.

We can nitpick the duration all you want, but we both know it does not take very long for this to occur. It happens particularly fast if you stray from the original topic and/or aren’t using a frontier model.

I am not going to call something “intelligent” when it can’t sustain a basic conversation.

Also, I would say SOTA LLM models can sustain a longer, more intelligent conversation than most people. Just in context length alone, they can keep track of more context than humans. But they're nowhere near as efficient the human brain, or the attention to quickly connect and find decades worth of memory like us.

To me AGI is all about the "G" general (we already had the AI part). General meaning universal, everything. It's not a function of knowledge or specific hardcoded tests, it's that you could give it a test it's never heard of before and never been trained on and it would ace it (it might need a lot of time).

Currently LLMs can't even really learn within a conversation, they can add a note to context and try to not drop it. Example things an AI cannot do yet (but maybe someday will):

- write a well-received book, write a best-seller

- come up with a new company idea, Run that company

- actually have a decent conversation, maybe someday talk somebody out of suicide effectively

- come up with its own ideas or theories that nobody else has presented

- understand the stock market well enough to trade better than an index fund

- be an expert Game Master in a TTRPG (making no mistakes, getting a read on the players' fantasies, calibrating difficulty in response to emotions)

- come up with a theory of what makes games fun, make a popular game

- be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)

- be able to articulate what it knows, what it doesn't know, and what information it would need to have to answer complex queries

- exhibit metacognition (thinking about its own thinking) and self-optimization

- wonder about things

- observe contradictions and ironies in the social-consciousness, do a standup routine that makes you rethink how you look at things

Some problems presented needs a very large context and some are not much solvable (e.g. trading) since market responds to traders' actions, as well, making it effectively an oracle problem (of computation).

On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them. However, brains in nature never stops. Wonder, daydream, sleep, self-evolve, clean up and eliminate memories and views and much more.

The hardware of human intelligence is not a singular thing that is uniform throughout. You cannot take the prefrontal cortex white matter out of someone's head and say you are holding a person. Much of the parts of our brains that enable much of our intelligence, is made of different specialized stuff. The visual cortex and sensorimotor regions aren't only there for input and output, they are used by the more thinky parts of the brain to do visualization and spatial reasoning. The cerebellum contains billions of neurons making little oscillator circuits and PID-like self-regulation machines that help make muscles do what they're supposed to, but also provide attention and time perception.

Heck, our brains contain language models, that train themselves up based on a glut of data over a span of about 10 years, and then they become more or less set in stone for the rest of our lives. Of course we can learn languages, but the "Critical Period" is a very real thing that produces a permanent architecture for some grammatical structures, or things like the ability to partition a lexicon by gender for faster lexical access which cannot be learned as an adult if your native language did not have gender.

I'm not trying to make a direct analogy, the point is that the language model doesn't need to be fully "generally intelligent" all on its own for there to exist a general intelligence, because the language model can be part of a generally intelligent system, which can do things like form, recall, and manage memories which are by now a standard feature in basically every chatbot.

Same with instruments. I started to play instruments at an early age, but started to play guitar around that age. Well, I'm not a virtuoso, but can play and more importantly can improve.

These AI systems we built are static things. We generally try to make them more intelligent by augmenting the context they can see, but the model doesn't evolve in every turn, for example.

Intelligence is a multi-faceted and multi-input construct, that's true, and GPT-6 may do amazing things w.r.t. other models, I didn't try it yet. OTOH, my main call is to remember that these are still static algorithms fed with enormous amount of data. They are more rooted on statistics rather than fixed inputs. In short, they are still fitting to the frame of "advanced search".

In general, I'm not against the tech, but the hype. I have other gripes about how AI is being built, but that's not subject of this comment.

The parent commenter noted:

"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"

Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.

Nature doesn't work that way.

Most intelligence researchers would agree that people seem to have a genetic cap on their intelligence. While someone can underperform their intellectual potential with an upbringing that doesn't adequately enrich their minds, it's near-impossible for humans to become more intelligent through reading, studying, etc.

When humans learn we gain knowledge, not intelligence.

I think the only real difference is that we humans are born lacking a lot of initial knowledge/data which means we have to go through a decade or more of education to reach our potential intelligence. LLMs on the other hand come pre-loaded with that knowledge.

Passed this point, wherever knowledge is passed in as context or stored in the neural net I don't think is that significant personally. I'm of course not suggesting we're exactly the same as LLMs and there is no noteable difference, I just don't think continual learning is as important as some suggest it is – at least assuming a model is deployed with adequate training such that it reaches its potential given it's size + architecture.

Machines aimed at running AGI are provided that, but the money spent there could as well go towards enhancing humans.

The claim is furthermore that truly original thinking, the infamous leaps in understanding and creativity, happen by attaining points outside such a convex hull.

It's hard to rigorously verify or disprove this claim. Hopefully this helps build an intuition of why the claim is not as shallow and obviously wrong as it may seem initially.

But yes, metaphorically I think that's right.

I think what OP was drawing a comparison to is that AI right now could not come up with an award winning novel from the spark of some creative notion and working up from there, as opposed to just mashing together what has already been done and calling it a day.

I agree, but in this field we value evidence. So there needs to be some test of novel-writing abilities.

Once there is, AI companies will be out to score highly on it.

Wait for a resurgence of Philip K Dick-style novels as humans desperately try to write things LLMs cannot.

I myself can't wait for Finnegan's Wake 2

I think if a person had those same advantages (e.g. could spend 5 hours thinking about what to say next) we could all hold outstanding conversations, or if we had read every book ever written I think many of us could write a very popular book, if we could read every singe company's P&L statement in a few seconds we could invest better than an index fund.

What I'm pointing out here is that these models appear to be intelligent when they really are simply unimagineably knowledgeable. When you drop the time-constraints it starts to become more and more apparent that human intelligence scales better with time than AI does (much in the same way AI can burp out tons of code but make your codebase entirely illegible within a matter of months).

Perhaps to simplify: my notion of intelligence is how much can you deduce with a constant set of starting context