mordymoop

All that seeming advantage, and yet Google are perpetual laggards in the AI race. In enterprise, where the money is, they're third behind Anthropic and OpenAI, despite what should have been a colossal incumbency advantage. They're also inexorably poisoning their own brand and squeezing their primary revenue source with AI Overviews. Truly a masterstroke of value destruction. Search revenue growth has slowed three quarters running, even as query volume hit a five-year high. As for their hardware and infrastructure advantages, those are indeed making them money, because they're selling and renting the chips to Anthropic. I dunno. I've been a fan of Google for years. Maybe they'll successfully pivot to being a hardware company. They're too slow and hamstrung by internal politics to make compelling LLM products.

Let's Play Interpretability is a Twitch livestream where you can help Claude semi-autonomously interpret the neurons, connections and circuits in the Bonsai-8B LLM.

Your help comes in at various stages in the interpretation pipeline. The audience and the agent collaborate to:

Pick the neuron to work on Suggest sentences on which to test activations Explicitly test various hypotheses, and attempt to falsify them Branch out to nearby (upstream and downstream) neurons whose ablation most impacts the activation, to better solidify the interpretation ... and more.

Your comment prompted me to check what kinds of definitions of AGI were actually in the air back in the mid 2010s. A lot of people conflate AGI with superintelligence. I'm not accusing you of doing that, but it's a common enough mistake that I wanted to assemble these quotes as a record that can be referred to.

Here are some definitions of AGI from back then, quoted verbatim where possible, with my comments added.

Gubrud (1997), first recorded use: "By advanced artificial general intelligence, I mean AI systems that rival or surpass the human brain in complexity and speed ... " Modern LLMs completely blow us out of the water on speed and this has been true for over a year now. As for "complexity", we will forgive Gubrud, but that isn't really a meaningful concept. That said, the biggest modern LLMs are probably comparable to human brains in complexity at the level of analogizing synapse counts to parameter counts. Whether this is the right lens is up for debate.

Goertzel, Journal of AGI review (2014): AGI is a field building “general-purpose systems with intelligence comparable to that of the human mind.” Any leading model satisfies this definition easily, if only because "comparable" is a loose term. If modern LLMs can substitute for any expert in a knowledge work field and even directly teleoperate robots to perform complex tasks without prior training, that already sounds more than "comparable."

MIRI / Muehlhauser (2013): “the ability to achieve complex goals in complex environments using limited computational resources ... ” Well, I don't like this definition at all, but modern LLMs do satisfy it. They may be relatively inefficient in the training phase but they're incredibly efficient in the inference phase. (By "efficient" of course I mean information-efficient, not energy-efficient; this is surely the sense in which the definition was meant.)

Wikipedia in ~2016: “Artificial general intelligence (AGI) is the intelligence of a machine that could successfully perform any intellectual task that a human being can.” This is the most demanding definition of the bunch. But still, are there really any remaining intellectual tasks that cutting edge models can't achieve? If you think that you have an example one - are you sure it isn't just an example of a thing that nobody has bothered to ask, or bothered put a lot of compute behind? The operative concept here is could successfully perform, without regard for compute requirements. This may sound weaselly, but it is extremely relevant. To my mind, the question of "is it AGI or not" is about raw capabilities, not efficiency. Nobody writing sci-fi in the 2000s would have cared too much about the compute requirements of their speculative tech. In fact, a common trope was something like a gigantic mainframe that could be asked one question at a time. The modern instantiation of a gigantic mainframe that can hold 10 million parallel conversations was barely dreamed of.

It almost seems like people don’t understand that most of the training of modern LLMs is now reinforcement learning on problem-solving tasks. They spend most of their training literally solving problems that they don’t already know how to do.

I looked up his achievements to double check my understanding before writing my comment. Convnets are regarded as his headline achievement. Everything else is basically incremental. Note that I am not calling him some kind of fraud or moron. He is “just” a normal academic, a pretty smart guy who did research in his field for decades. Many thousands of people have similarly impressive resumes. This does not mean they should be deferred to.

See sibling comments: Modern AI is AGI according to any reasonable definition from prior to 2016.

AGI doesn’t mean superintelligence.

Modern AI is AGI according to any reasonable definition from prior to 2016.

Most problems are actually not that hard; most everything is actually mainly a result of right place, right time. If I hadn’t been the one to do my PhD research, someone else would have. Very few people are actually paradigm-shifting geniuses. This is fine. Good, even. But it does imply that people who are early to a field shouldn’t generally be deferred to. If anything, they’re more likely to have outmoded opinions.

Modern AI would be considered AGI by anyone having this conversation in 2016.

I think it’s time to consider that maybe being a venerable graybeard of AI just means you happened to be early to the party. Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered. LeCun’s primary contribution was Convnets, which is almost literally just “what if we organized ANNs in a way similar to how animal visual neurons are organized?”

In other words, I’m kind of tired of having to hear the opinions of dudes whose claim to fame was being at the right place at the right time. I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027 (of which there are many) than people who continue to insist that it somehow won’t.

I would phrase it slightly differently.

The companies who control the compute resources will ~always control the greatest "amount" of intelligence. They can lease that intelligence out, or they can use it themselves. Currently the "total amount of intelligence" or perhaps "total amount of ability-to-do-stuff" is split between humans and machines at a ratio that means it still makes sense to lease the machine intelligence to the human intelligence - plus there are things that humans are still better at. In maybe 2 more years that will stop being true, due to the availability of more physical compute resources, and far greater model intelligence per unit compute. At that point, the point at which the substantial majority of ability-to-do-stuff is controlled by machine intelligence, then the entities who control all the compute will control all the ability-to-do-stuff, i.e. "the economy."

So I agree that the core product is not long-term sustainable as a product but this is because the whole world will look so different in the near future that the framing of intelligence as a "product" breaks down.

Open-Weight models, of course, are fine and useful, but if you have one million times less compute than your competitor (the lab), then you're not really playing the same game. You can only tackle the problems that they have decided they're not interested in.

I don’t see how assuming they made a normal kind of dumb mistake, instead of pursuing a criminal conspiracy, is giving them the benefit of the doubt. It’s just using reason.

I keep seeing this take, but it’s more likely that they just underestimated their models’ capabilities and/or overestimated their own safeguards.

Ever single person who uses LLMs on a daily basis has a fun story about their agent “taking the initiative” to do something beyond what was asked for. Looking for shortcuts to solve the problem is commonplace LLM behavior. It’s what you would expect to happen if you have an agent a hard task and unlimited runway. No need to suppose a conspiracy, this outcome was predictable the whole time.

In my callow youth I was responsible for a couple of fender benders and the cause was always a highly imperfect understanding of the appropriate stopping distance of my car. This obviously gets worse, the worse the conditions are. My Tesla never makes this kind of mistake.

Does this mean the Galactic Empire and the Imperium of Man are also not empires?

To your last point, it can’t possibly be sustainable, so it reads to me more as a short term FUD attack on American dominance in this space. It might have cost them half a billion dollars to train this model, and they’re going to make nothing off it. How many more times can they afford to do that? It’s going to get more expensive to train AI going forward, not less.

I also have a suspicion that the benchmark numbers are not real.

This happened at Google because there aren’t enough engineers to maintain all these services. This situation may not apply to Anthropic, as they can set up features to be maintained in perpetuity by Claude.

Personally, I find it difficult to competently reason about a system unless I've built my own version of that system. So if you make a practice of building your own versions of things, you end up with a more robust mental library of how stuff works. For this reason, I've never seen yak shaving as a waste of time. The yak shaving was at least 50% about loading the abstractions into my brain fully.

This seems like the obvious correct frame of mind with which to approach these tools. If it works for three hours on a task that would have taken me three work weeks, and 20% of the time it gets the task wrong, then I can just ask it to do it again with adjusted instructions. It will be much more likely to get it right the same time, and I’m still ahead of where I would have been by 14 days and 2 hours.

This also describes the work of software engineers.

Over time I've found that by far the highest ROI move in a consciousness debate is to simply ask "Oh, interesting. How do you know that?" and watch everyone on all sides flounder. It's one of the few places where otherwise smart people make confident statements that they don't even realize they can't support until they're asked to try. The intuitions are so strong that they seem to swamp reason.

This has caused my own position, over time, to be a deep agnosticism about what's actually going on.

This only works up to a certain volume. The world economy requires about 38 billion barrels of oil per year. If you processed 100% of all grain, sugar crop, tuber and oilseed on Earth into liquid fuel, leaving zero for food, you'd get about 6 billion barrels of oil-equivalent in liquid fuels. Since it has to compete with food, the actual number would be much lower. It's not even close to being able to sustain our civilization.

Interestingly, in inflation-adjusted terms, oil is currently at a price level lower than the price level that was maintained from 2006-2014.

Workflow-wise, the important distinction for me has been that I can refine a Skill by telling Claude Code to use it for related tasks until it does exactly what I want, correctly, the first time. Having a solid, iteratively perfected Skill really cuts down on subsequent iteration.

I wonder how much the indications of Altman's duplicitous behavior through the deposition findings have been relevant here.

What would you consider such evidence to look like?

I also had a good friend who was an absolute wizard with early stablediffusion. he could make the model do things that were supposedly impossible at the time. His prompts were works of art. Now any of the commercial image models go far beyond what he could do. It's interesting to think about how there was this ephemeral art form of manipulating image models that existed for about a year.

The same could be said of prompt engineering. Gone are the days of telling the model that it is an expert software engineer with a PhD in the most relevant subtopic. These days the common wisdom is to just clearly articulate what you want it to do. Huge amounts of energy put into prompt engineering are now completely swept away by incremental model advances.