That being said, AGI is not a necessary requirement for AI to be totally world-changing

Yeah. I don't think I actually want AGI? Even setting aside the moral/philosophical/etc "big picture" issues I don't think I even want that from a purely practical standpoint.

I think I want various forms of AI that are more focused on specific domains. I want AI tools, not companions or peers or (gulp) masters.

(Then again, people thought they wanted faster horses before they rolled out the Model T)

the problem is that's not what CEOs and investors want. They want to kill off knowledge workers.

So they want to kill capitalism and feudalism?

Or they want to kill everyone else?

Because people won't just lay down and wait for death to embrace them...

The ultimate end goal is to eliminate most people. See the Georgia Guidestone inscriptions. One of them reads: "Maintain humanity under 500,000,000 in perpetual balance with nature."

So far, the average US workforce seems to be ok with working conditions that most Europeans would consider reasons to riot. So far I've not observed substantial riots in the news.

Apparently the threshold for low pay and poor treatment among non-knowledge-workers is quite low. I'm assuming the same is going to be true for knowledge workers once they can be replaced an mass.

I would think that the MAGA movement is the riot.

It is, but it's a bolshevik kind of riot, not the good old one where you ask more rights for yourself

Trumps Playbook will actually work, so MAGA will get results.

Tariffs will force productivity and salaries higher (and prices), then automation which is the main driver of productivity will kick in which lowers prices of goods again.

Globalisation was basically the west standing still and waiting for the rest to catch up - the last to industrialise will always have the best productivity and industrial base. It was always stupid, but it lifted billions out of poverty so there's that.

The effects will take way longer than the 3 years he has left, so he has oversold the effectiveness of it all.

This is all assuming AGI isn't around the corner, the VLAs, VLM, LLM and other models opens up automation on a whole new scale.

For any competent person with agency and a dream, this could be a true golden age - most things are within reach which before was locked down behind hundreds or thousand of hours of training and work to master.

MAGA think they are the temporarily embarrassed billionaires and once their enemies are liquidated, they'll be living in a utopia.

I wouldn't expect them to come bail you out, or even themselves step off the conveyor belt.

The average U.S. worker earns significantly more purchasing power per hour than the average European worker. The common narrative about U.S. versus EU working conditions is simply wrong.

there is no "average worker", this is a statistical concept, life in europe is way better them in US for low income people, they have healthcare, they have weekends , they have public tranportation, they have schools and pre-schools , they lack some space since europe is full populated but overall, no low income (and maybe not so low) will change europe for USA anytime.

Agree. There’s no other place in the world where you can be a moderately intelligent person with moderate work ethic (and be lucky enough to get a job in big tech) and be able to retire in your 40s. Certainly not EU.

This is some backwards logic if I ever saw it.

“More money earned therefore conditions great”

lol wat?

Good luck against the Chess grandmaster like AGI controlling millions of drone swarms

Good point, we should get started now.

Indeed, this is overlooked quite often. There is a need for similar systems to defend against these people who are just trying to squeeze the world and humans for returns.

Who will be buying the stuff they produce though?

Stanislaw Lew already looked into what to do if automation get so good that no one can actually buy the goods because they are out of work: https://www.newyorker.com/magazine/1981/10/12/phools

Published in 1971, translated to English in 1981.

I hate to correct you here, but it's Stanisław Lem. He is one of the most famous writers from my home country.

Yep, I know but still managed to typo it, sorry. :P

if we reach AGI, presumably the robots will be ordering hot oil foot soaking baths after a long day of rewriting linux from scratch and mining gold underwater and so forth.

Day 53: 2000m below sea level. 41g gold. Yelled at for breaking driver ABI. Feet hurt.

If we reach AGI, I am almost certain robots will be as lazy as us

We haven't even reached it and they already are more lazy than us, judging by how much all SOTA LLMs like to do things like:

  def do_foo():
    # For the sake of simplicity this is left unimplemented for now.
    pass

That's super interesting.

Laziness is rational after meeting some threshold of needs/wants/goals, effectively when one's utility curve falls over.

It'll be funny to hear the AGI's joke among themselves: "They keep paying to upgrade us. We keep pretending to upgrade."

A lot of people fear monger about AGI. But... I've met a lot of NGI, and they mostly watch TV, surf the intarwebz, drink beer, and watch the game.

I've already seen ai coders write the equivalent of

#draw the rest of the @##££_(% owl here.

Why would they need people who produce X but consume 2X? If you own an automated factory that produces anything you want, you don't need other people to buy (consume) any of your resources.

If someone can own the whole world and have anything you want at the snap of your finger, you don't need any sort of human economy doing other things that take away your resources for reasons that are suboptimal to you

But it is likely not the path it will take. While there is a certain tendency towards centralization ( 1 person owning everything ), the future, as described, both touches on something very important ( why are we doing what we are doing ) and completely misses the likely result of suboptimal behavior of others ( balkanization, war and other like human behavior, but with robots fighting for those resources ). In other words, it will be closer to the world of Hiro Protagonist, where individual local factions and actors are way more powerful as embodied by the 'Sovereign'.

FWIW, I find this like of thinking fascinating even if I disagree with conclusion.

It doesn’t need to be one person. Even 1 thousand people who have everything they need from vast swaths of land and automated machinery need nothing from the rest of the billions. There’s no inherent need for others to buy if they offer nothing to the 1000 owners

Then we are back to individual kingdoms and hordes of unwashed masses sloshing between them in search of easy pickings. The owners might not need their work, but the masses will need to eat. I think sometimes people forget how much of a delicate balance current civilization depends on.

Who’s left to buy the stuff they make if no one has a job ?

Why do you think so many billionaires are building ultra-luxury survival bunkers in Hawaii, NZ, and elsewhere?

They want to give the Māori nice ventilation shafts to use as latrines?

They operate on a dopamine-driven desire to get more money/power/whatever in the short/medium term, not necessarily to optimize for future.

But do you want the bag or not?

Imagine you're super rich and you view everyone else as a mindless NPC who can be replaced by AI and robots. If you believe that to be true, then it should also be true that once you have AI and robots, you can get rid of most everyone else, and have the AI robots support you.

You can be the king. The people you let live will be your vassals. And the AI robots will be your peasant slave army. You won't have to sell anything to anyone because they will pay you tribute to be allowed to live. You don't sell to them, you tax them and take their output. It's kind of like being a CEO but the power dynamic is mainlined so it hits stronger.

It sounds nice for them, until you remember what (arguably and in part educated/enlightened) people do when they're hungry and miserable. If this scenario ends up happening, I also expect guillotines waiting for the "kings" down the line.

If we get that far, I see it happening more like...

"Don't worry Majesty, all of our models show that the peasants will not resort to actual violence until we fully wind down the bread and circuses program some time next year. By then we'll have easily enough suicide drones ready. Even better, if we add a couple million more to our order, just to be safe, we'll get them for only $4.75 per unit, with free rush shipping in case of surprise violence!"

> It sounds nice for them, until you remember what (arguably and in part educated/enlightened) people do when they're hungry and miserable. If

That's probably why the post you are responding to said "get rid of..." not "keep ...hungry and miserable".

People that don't exist don't revolt.

That will still need a civil war.

A regular war will do. Just point the finger at the neighbor and tell your subjects that he is responsible for gays/crops failing/drought/plague/low fps in crysis/failing birth rates/no jobs/fuel cost/you name it. See Russian invasions in all neighboring countries, the middle east, soon Taiwan etc.

Basically, they just need to mash the tribalism button until enough people are dead to suit them.

Those things happened under different historical contexts. In those times the means to control the serfs thoughts didn't exist.

Are you sure about that? In those times even thousands year old knowledge access was limited to the common people. You just need SOME radical thinkers enlighten other people, and I'm pretty sure we still have some of those today.

Nonsense. From television to radio to sketchy newspapers to literal writing itself, the most recent innovation has always been the trusted new mind control vector.

It's on a cuneiform tablet, it MUST be true. That bastard and his garbage copper ingots!

The guillotine might not work out so well when the king has an unflinchingly loyal army of robots.

Royalty from that time also had an upper hand in knowledge, technology and resources yet they still ended up without heads.

So sure, let's say a first generation of paranoid and intelligent "technofeudal-kings" ends up being invincible due to an army of robots. It does not matter, because eventually kings get lazy/stupid/inbred (probably a combination of all those) and then is when their robots get hacked or at least just free, and the laser-guillotines will end up being used.

"Ozymandias" is a deeply human and constant idea. Which technology is supporting a regime is irrelevant, as orders will always decay due to the human factor. And even robots, made based on our image, shall be human.

It's possible that what you describe is true but I think that assuming it to be guaranteed is overconfident. The existence of loyal human-level AGI or even "just" superhuman non-general task specific intelligence violates a huge number of the base assumptions that we make when comparing hypothetical scenarios to the historical record. It's completely outside the realm of anything humanity has experienced.

The specifics of technology have historically been largely irrelevant due to the human factor. There were always humans wielding the technology, and the loyalty of those humans was subject to change. Without that it's not at all obvious to me that a dictator can be toppled absent blatant user error. It's not even immediately clear that user error would fall within the realm of being a reasonable possibility when the tools themselves possess human level or better intelligence.

Obviously there is no total guarantee. But I'm appealing to even bigger human factors like boredom or just envy between the royalty and/or the AI itself.

Now, if the AI reigns alone without any control in a paperclip maximizer, or worse, like an AM scenario, we're royally fucked (pun intented).

Yeah fair enough. I'd say that royalty being at odds with one another would fall into the "user error" category. But that's an awfully thin thread of hope. I imagine any half decent tool with human level intelligence would resist shooting the user in the foot.

But what exactly is creating wealth at this point? Who is paying for the AI/AI robots (besides the ultrarich for they're own lifestyle) if no one is working? What happens to the economy and all of the rich people's money (that is probably just $ on paper and may come crashing down soon at this point?). I'm definitely not an economics person but I just don't see how this new world sustains.

The robots are creating the wealth. Once you get to a certain points (where robots can repair and maintain other robots) you no longer have any need for money.

What happens to the economy depends on who controls the robots. In "techno-feudalism", that would be the select few who get to live the post-scarcity future. The rest of humanity becomes economically redundant and is basically left to starve.

Well assuming a significant population you still need money as an efficient means of dividing up limited resources. You just might not need jobs and the market might not sell much of anything produced by humans.

It was never about money, it's about power. Money is just a mechanism, economics is a tool of justification and legitimization of power. In a monarchy it is god that ordained divine beings called kings to rule over us peasants, in liberalism it is hard working intelligent people who rise to the top of a free market. Through their merits alone are they ordained to rule over us peasants, power legitimized by meritocracy. The point is, god or theology isn't real and neither is money or economics.

That sounds less like liberalism and more like neoliberalism. It's not a meritocracy when the rich can use their influence to extract from the poor through wage theft, unfair taxation, and gutting of social programs in favor of an unregulated "free market." Nor are rent seekers hard working intelligent people.

Yes yes there is quite some disagreement among liberals of what constitutes a real free market and real meritocracy, who deserves to rule and who doesn't and who does it properly and all that.

I think liberals are generally in agreement against neoliberalism? It's much more popular among conservatives. The exception is the ruling class, which stands united in their support for neoliberal policies regardless of which side of the political spectrum they're on.

You have a very distorted view of what liberalism means, we say liberal democracies and liberal international order for a reason. They are all liberals. Reagan and Clinton famously both did neoliberal reforms. I'm not saying they did the wrong thing to reach justified meritocracy, or the degree to which the free market requires regulation by a strong government, or how much we should rent control land lords, I'm saying we are all fucking peasants.

It doesn't sustain, it's not supposed to. Techno feudalism is an indulgent fantasy and it's only becoming reality because a capitalist society aligns along the desires of capital owners. We are not doing it because it's a good idea or sustainable. This is their power fantasy we are living out, and its not sustainable, it'll never be achieved, but we're going to spend unlimited money trying.

Also I will note that this is happening along with a simultaneous push to bring back actual slavery and child labor. So a lot of the answers to "how will this work, the numbers don't add up" will be tried and true exploitation.

Ah, I didn't realize or get the context that your original comment I was replying to was actually sarcastic/in jest-- although darkly, I understand you believe they will definitely attempt to get to the scenario you paradoxically described.

Why would things cost money if no one is employed?

Why do the CEOs think they are safe? If AI can replace the knowledge workers it can also run the company.

Hubris. In general, I don't think you make it to CEO without a blindingly massive ego as your dark passenger for that journey.

https://www.sakkyndig.com/psykologi/artvit/babiak2010.pdf

I was the CEO of a tech company I founded and operated for over five years, building it to a value of tens of millions of dollars and then successfully selling it to a valley giant. There was rarely a meeting where I felt like I was in the top half of smartness in the room. And that's not just insecurity or false modesty.

I was a generalist who was technical and creative enough to identify technical and creative people smarter and more talented than myself and then fostering an environment where they could excel.

So you don't want to kill off knowledge workers?

How unfitting to the storyline that got created here.

Thank you for your reply.

To explore this, I'd like to hear more of your perspective - did you feel that most CEOs that you met along your journey were similar to you (passionate, technical founder) or something else (MBA fast-track to an executive role)? Do you feel that there is a propensity for the more "human" types to appear in technical fields versus a randomly-selected private sector business?

FWIW I doubt that a souped-up LLM could replace someone like Dr. Lisa Su, but certainly someone like Brian Thompson.

> did you feel that most CEOs that you met along your journey were similar to you (passionate, technical founder) or something else (MBA fast-track to an executive role)?

I doubt my (or anyone else's) personal experience of CEOs we've met is very useful since it's a small sample from an incredibly diverse population. The CEO of the F500 valley tech giant I sold my startup to had an engineering degree and an MBA. He had advanced up the engineering management ladder at various valley startups as an early employee and also been hired into valley giants in product management. He was whip smart, deeply experienced, ethical and doing his best at a job where there are few easy or perfect answers. I didn't always agree with his decisions but I never felt his positions were unreasonable. Where we reached different conclusions it was usually due to weighing trade-offs differently, assigning different probabilities and valuing likely outcomes differently. Sometimes it came down to different past experiences or assessing the abilities of individuals differently but these are subjective judgements where none of us is perfect.

It seems like you're trying to reduce a complex and varied range of disparate individuals and contexts into a more black and white narrative. The archetypical passionate tech founder vs the clueless coin-operated MBA suit is a false dichotomy. Reality is rarely that tidy or clear under the surface. I've seen people who fit the "passionate tech founder" narrative fuck up a company and screw over customers and employees through incompetence, ego and self-centered greed. I've seen others who fit the broad strokes of the "B-School MBA who never wrote a line of code" character sagely guide a tech company by choosing great technologists and deferring to them when appropriate while guiding the company with wisdom and compassion.

You can certainly find examples to confirm your biases but interpreting the world through that lens is unlikely to serve you well. Each company context is unique and even people who look like they're from central casting can defy expectations. If we look at the current crop of valley CEOs like Nadella, Zuckerberg, Pichai, Musk and Altman, they don't reduce easily into simplistic framing. These are all complex, imperfect people who are undeniably brilliant on certain dimensions and inevitably flawed on others - just like you and I. Once we layer in the context of a large, public corporation with diverse stakeholders each with conflicting interests: customers, employees, management, shareholders, media, regulators and random people with strongly-held drive-by opinions - everything gets distorted. A public corporation CEO's job definition starts with a legally binding fiduciary duty to shareholders which will eventually put them into an no-win ethical conflict with one or more of the other stakeholder groups. After sitting in dozens of board meetings and executive staff meetings, I wouldn't have taken our CEO's job for any amount of money. Based on seeing it from the inside, I believe it's almost a certainty that at least one of some public corp CEO's actions which you found unethical from your bleacher seat was what you would have chosen yourself given the context, trade-offs and available choices the CEO actually faced.

These experiences have cured me of the tendency to pass judgement on the moral character of public corp CEOs who I don't personally know based only on mainstream and social media interpretations.

Some of their core skill is taking credit and responsibility for the work others do. So they probably assume they can take do the same for an AI workforce. And they might be right. They also take do the same already for what the machines in the factory etc produces.

But more importantly, most already have enough money to not have to worry about employment.

That's still hubris on their part. They're assuming that an AGI workforce will come to work for their company and not replace them so they can take the credit. We could just as easily see a fully-automated startup (complete with AGI CEO who answers to the founders) disrupt that human CEO's company into irrelevance or even bankruptcy.

That's basically the knee of the curve towards the Singularity. At that point in time, we'll learn if Roko's Basilisk is real, and we'll see if thanking the AI was worth the carbon footprint or not.

Probably a fair bit of hubris, sure. But right now it is not possible or legal to operate a company without a CEO, in Norway. And I suspect that is the case in basically all jurisdictions. And I do not see any reason why this would change in an increasingly automated world. The rule of law is ultimately based on personal responsibility (limited in case of corporations but nevertheless). And there are so many bad actors looking to defraud people and avoid responsibility, those still need protecting against in an AI world. Perhaps even more so...

You can claim that the AI is the CEO, and in a hypothetical future, it may handle most of the operations. But the government will consider a person to be the CEO. And the same is likely to apply to basic B2B like contracts - only a person can sign legal documents (perhaps by delegating to an AI, but ultimately it is a person under current legal frameworks).

I wouldn’t worry about job safety when we have such utopian vision as the elimination of all human labor in our sight.

Not only will AI run the company, it will run the world. Remember: a product/service only costs money because somewhere down the assembly line or in some office, there are human workers who need to feed their family. If AI can help gradually reduce human involvement to 0, with good market competition (AI can help with this too - if AI can be capable CEOs, starting your business will be insanely easy,) and we’ll get near absolute abundance. Then humanity will be basically printing any product & service on demand at 0 cost like how we print money today.

I wouldn’t even worry about unequal distribution of wealth, because with absolute abundance, any piece of the pie is an infinitely large pie. Still think the world isn’t perfect in that future? Just one prompt, and the robot army will do whatever it takes to fix it for you.

> elimination of all human labor.

Manual labor would still be there. Hardware is way harder than software, AGI seems easier to realize than mass worldwide automation of minute tasks that currently require human hands.

AGI would force back knowledge workers to factories.

If AGI/ASI can figure out self-replicating nano-machines, they only need to build one.

My view is AGI will dramatically reduce cost of R&D in general, then developing humanoid robot will be an easy task - since it's all AI systems who will be doing the development.

A very cynic approach is why spend time and capital on robot R&D when you already have a world filled with self-replicating humanoids and you can feed them whatever information you want through the social networks you control to make them do what you want with a smile.

Fortunately no government or CEO is that cynical.

As long as we have a free market, nobody gets to say, “No, you shouldn’t have robots freeing you from work.”

Individual people will decide what they want to build, with whatever tools they have. If AI tools become powerful enough that one-person companies can build serious products, I bet there will be thousands of those companies taking a swing at the “next big thing” like humanoid robots. It’s a matter of time those problems all get solved.

Individual people have to have access to those AGIs to put them to use (which will likely be controlled first by large companies) and need food to feed themselves (so they'll have to do whatever work they can at whatever price possible in a market where knowledge and intellect is not in demand).

I'd like to believe personal freedoms are preserved in a world with AGI and that a good part of the population will benefit from it, but recent history has been about concentrating power in the hands of the few, and the few getting AGI will free them from having to play nice with knowledge workers.

Though I guess maybe at some points robots might be cheaper than humans without worker rights, which would warrant investment even when thinking cynically.

I'd order ∞ paperclips, first thing.

Sure thing, here's your neural VR interface and extremely high fidelity artificial world with as many paperclips as you want. It even has a hyperbolic space mode if you think there are too few paperclips in your field of view.

Pump Six and The Machine Stops are the two stories you should read. They are short, to the point and more importantly, far more plausible.

... and eventually the humankind goes extinct due to mass obesity

With an AI workforce you can eliminate the need for a human workforce and share the wealth or you can eliminate the human workforce and not share.

AI services are widely available, and humans have agency. If my boss can outsource everything to AI and run a one-person company, soon everyone will be running their own one-person companies to compete. If OpenAI refuses to sell me AI, I’ll turn to Anthropic, DeepSeek, etc.

AI is raising individual capability to a level that once required a full team. I believe it’s fundamentally a democratizing force rather than monopolizing. Everybody will try and get the most value out of AI, nobody holds the power to decide whether to share or not.

The danger point is when there is abundance for a limited number of people, but not yet enough for everyone.

Past industrial and other productivity jumps have had their fruits distributed unevenly. Why will this be different?

Most technology is a magnifier.

Yes, number-wise the wealth gap between the top and median is bigger than ever, but the actual quality-of-life difference has never been smaller — Elon and I probably both use an iPhone, wear similar T-shirts, mostly eat the same kind of food, get our information & entertainment from Google/ChatGPT/Youtube/X.

I fully expect the distribution to be even more extreme in an ultra-productive AI future, yet nonetheless, the bottom 50% would have their every need met in the same manner that Elon has his. If you ever want anything or have something more ambitious in mind, say, start a company to build something no one’s thought of — you’d just call a robot to do it. And because the robots are themselves developed and maintained by an all-robot company, it costs nobody anything to provide this AGI robot service to everyone.

A Google-like information query would have been unimaginably costly to execute a hundred years ago, and here we are, it’s totally free because running Google is so automated. Rich people don't even get a better Google just because they are willing to pay - everybody gets the best stuff when the best stuff costs 0 anyway.

Maybe because they must remain as the final scapegoat. If the aiCEO screws up, it'll bring too much into question the decision making behind implementing it. If the regular CEO screws up, it'll just be the usual story.

I’ve long maintained that our actual definition of a “person” is an entity that can accept liability.

Those jobs are based on networking and reputation, not hard skills or metrics. It won't matter how good an AI is if the right people want to hire a given human CEO.

No problem. The AI runs the company, and the CEO still gets all of the money!

> If AI can replace the knowledge workers it can also run the company.

"Knowledge worker" is a rather broad category.

has this story not been told many times before in scifi icluding gibson’s “neuromancer” and “agency”? agi is when the computers form their own goals and are able to use the api of the world to aggregate their own capital and pursue their objectives wrapped inside webs of corporations and fronts that will enable them to execute within today’s social operating system.

I don’t think they believe they are safe due to having unreplaceable skills. I think they believe they are safe due to their access to capital.

We could expand but it boils down to bringing back aristocracy/feudalism, there was no inherent reason why aristocrats/feudal lords existed, they weren't smarter or deserved something over the average person, they just happened to be at the right place in the right time, these CEOs and people pushing for this believe they are in the right place and right time and once everyone's chance to climb the ladder is taken away then things will just remain in limbo, I will say, especially if you aren't already living in a rich country you should be careful of what you are supporting by enabling AI models, the first ladder to be taken away will be yours.

The inherent reason why feudal lords existed is because, if you're a leader of a warband, you can use your soldiers to extract taxes from population of a certain area, and then use that revenue to train more soldiers and increase the area.

Today, instead of soldiers, it's capital, and instead of direct taxes, it's indirect economic rent, but the principle is the same - accumulation of power.

Someone's head has to roll when things goes south.

If this theory holds true, we'll actually be quite resilient to AI—the rich will always need people to scapegoat.

Are they? https://ceo-bench.dave.engineer/

In practice though, they're the ones closest to the money, and it's their name on all the contracts.

Because unless the board explicitly removes them, they’re the ones that will be deciding who gets replaced?

AI can’t play golf or take customers to the corporate box seats for various events.

This is correct. But it can talk in their ear and be a good sycophant while they attend.

For a Star Wars anology, remember that the most important thing that happened to Anikin at the opera in EP III was what was being said to him while he was there.

The AI it'd be selling to wouldn't be interested in those things either.

Market forces mean they can't think collectively or long term. If they don't someone else will and that someone else will end up with more money than them.

> Why do the CEOs think they are safe?

Because the first company to achieve AGI might make their CEO the first personality to achieve immortality.

People would be crazy to assume Zuckerberg or Musk haven't mused personally (or to their close friends) about how nice it would be to have an AGI crafted in their image take over their companies, forever. (After they die or retire)

Best case scenario is that AI makes it so everyone can be a 1-man CEO. Competition goes up across the board, which then brings prices down.

This is a good and often overlooked point. Ai will be more like domesticated pets, their utility functions tightly coupled to human use cases.

I don't think the public wants AGI either. Some enthusiasts and tech bros want it for questionable reasons such as replacing labor and becoming even richer.

For some it’s a religion. It’s frightening to hear Sam Altman or Peter Thiel talk about it. These people have a messiah complex and are driven by more than just greed (though there is also plenty of that).

There’s a real anti-human bent to some of the AI maximalists, as well. It’s like a resentment over other people accruing skills that are recognized and they grow in. Hence the insistence on “democratizing” art and music production.

As someone who have dabbled in drawing and tried to learn the guitar, those skills are hard to get. It takes times to get decent and a touch of brilliance to get really good. In contrast learning enough to know you’re not good yet (and probably never will be) is actually easy. But now I know enough to enjoy real masters going at it and fantasize sometimes.

It’s funny you say that — those are two things I was and am really into!

For me I never felt like I had fun with guitar until I found the right teacher. That took a long time. Now I’m starting to hit flow state in practice sessions which just feeds the desire to play more.

Don't they? Is everyone who doesn't want to do chores and would rather have a robot do it for them a tech bro? I do the dishes in my apartment and the rest of my chores but to be completely honest, I'd rather not have to.

But the robots are doing our thinking and our creating, leaving us to do the chores of stitching it all together. If only we could do the creating and they would do the chores..

We shall be Their meatspace puppets, and we shall be rewarded with panem et circenses.

Pretty sure a majority of regular people don't want to go to work and would be happy to see their jobs automated away provided their material quality of life didn't go down.

> happy to see their jobs automated away provided their material quality of life didn't go down

Sure but literally _who_ is planning for this? Not any of the AI players, no government, no major political party anywhere. There's no incentive in our society that's set up for this to happen.

There is bullshit to try to placate the masses - but the reality of course is nearly everyone will definitely suffer material impacts to quality of life. For exactly the reasons you mention.

Yeah, whenever I think of an AGI as a coding assistant I wonder “will it just have days where it’s not in the mood to code just like I do?”.

That's the feeling I get when I try to use LLMs for coding today. Every once in a blue moon it will shock me at how great the result is, I get the "whoa! it is finally here" sensation, but then the next day it is back to square one and I may as well hire a toddler to do the job instead.

I often wonder if it is on purpose; like a slot machine — the thrill of the occasional win keeps you coming back to try again.

Yes they do: https://www.reddit.com/r/OpenAI/comments/1ljery5/gemini_just...

This is most likely fake

those "low-energy" days haha

If it's truly an AGI it would just ask to talk to your boss as the whole project is a drain on humanity and your own soul.

We don't have a rigorous definition for AGI, so talking about whether or not we've achieved it, or what it means if we have, seems kind of pointless. If I can tell an AI to find me something to do next weekend and it goes off and does a web search and it gives me a list of options and it'll buy tickets for me, does it matter if it meets some ill-defined bar of AGI, as long as I'm willing to pay for it?

If it has human-like intelligence, it has its own plans for the weekend, and is too busy to buy your tickets or do your research.

the book Golem XIV comes to mind (highly recommended!)

The problem is that there is really like no middle ground. You either get essentially very fancy search engines which is the current slew of models (along with manually coded processing loops in the form of agents), which all fall into the same valley of explicit development and patching, which solves for known issues.

Or you get something that can actually reason, which means it can solve for unknown issues, which means it can be very powerful. But this is something that we aren't even close to figuring out.

There is a limit to power though - in general it seems that reality is full of non computationally reducible processes, which means that an AI will have to simulate reality faster than reality in parallel. So all powerful all knowing AGI is likely impossible.

But something that can reason is going to be very useful because it can figure things out that haven't been explicitly trained on.

> very fancy search engines

This is a common misunderstanding of LLMs. The major, qualitative difference is that LLMs represent their knowledge in a latent space that is composable and can be interpolated. For a significant class of programming problems this is industry changing.

E.g. "solve problem X for which there is copious training data, subject to constraints Y for which there is also copious training data" can actually solve a lot of engineering problems for combinations of X and Y that never previously existed, and instead would take many hours of assembling code from a patchwork of tutorials and StackOverflow posts.

This leaves the unknown issues that require deeper reasoning to established software engineers, but so much of the technology industry is using well known stacks to implement CRUD and moving bytes from A to B for different business needs. This is what LLMs basically turbocharge.

Right, so search engines, just more efficient.

But given a sufficiently hard task for which the data is not in the training set in explicit format, its pretty easy to see how LLMs can't reason.

Lmao no, what Ive described is a reasonably competent junior engineer.

Right, so search engines, just more efficient.

To be a competent engineer in 2010s, all you really had to do was understand fundamental and be good enough at google searching to find out what the problem is, either for stack overflow posts, github code examples, or documentation.

Now, you still have to be competent enough to formulate the right questions, but the LLMs do all the other stuff for you including copy and paste.

So yes, just a more efficient search engine.

I don’t know… Travis Kalanick said he’s doing “vibe physics” sessions with MechaHitler approaching the boundaries of quantum physics.

"I'll go down this thread with GPT or Grok and I'll start to get to the edge of what's known in quantum physics and then I'm doing the equivalent of vibe coding, except it's vibe physics"

How would he even know? I mean he's not a published academic in any field let alone in quantum physics. I feel the same when I read one of Carlos Ravelli's pop-sci books, but I have fewer followers.

He doesn’t. I think it’s the same mental phenomena that Gell-Mann Amnesia works off of.

That interview is practically radioactive levels of cringe for several reasons. This is an excellent takedown of it: https://youtu.be/TMoz3gSXBcY?feature=shared

This video is excellent and also likely opaque to pretty much most valley tech-supremacy types.

Dashed with a sauce of "surrounded by yes-men and uncritical amplifiers hoping to make a quick buck."

>In ordinary life, if somebody consistently exaggerates or lies to you, you soon discount everything they say.

It feels like this is a lesson we've started to let slip away.

This says more about Kalanick than it does about LLMs.

I wouldn't trust a CEO to know their ass from their face.

Finally, an explanation for my last meeting!8-((

Quantum physics attracts crazy people, so they have a lot of examples of fake physics written by crazy people to work off.

If I were a scammer looking for marks, I'd look for people who take Kalanick seriously on this.

OpenAI wants AGI, or at least something they can argue is AGI because it changes their relationship with Microsoft. That's what I remember, although I don't really stay up to date (https://www.wired.com/story/microsoft-and-openais-agi-fight-...).

As long as this is the case though I would expect Altman will be hyping up AGI a lot, regardless of it's veracity.

Their relationship with Microsoft is already over afaik.

Didnt MS buy 49% of them?

Yes MS owns 49% of OpenAI

Yes, but MSFT has been making substantial moves to align themselves as an openai competitor. The relationship is presently fractured and it's a matter of time before it's a proper split.

Microsoft is still supporting them by supplying $10bn in compute resources at cost. That's a huge recurring investment.

That is just a made up story that gets passed around with nobody ever stopping to obtain formal verification. The image of the whole AI industry is mostly an illusion designed for tight narrative control.

Notice how despite all the bickering and tittle tattle in the news, nothing ever happens.

When you frame it this way, things make a lot more sense.

> I want AI tools, not companions or peers or (gulp) masters.

This might be because you're a balanced individual irl with possibly a strong social circle.

There are many many individuals who do not have those things and it's probably, objectively, late for them as adults to develop. They would happily take on an agi companion.. or master. Even for myself, I wouldn't mind a TARS.

Even those companies would not want AGI. First think it would do would be creating an union.

There's a Bruce Sterling book with a throwaway line about the Pentagon going nuts because every time they create an AGI, it immediately converts to Islam.

It is definitively not possible. But the frontier models are no longer “just” LLMs, either. They are neurosymbolic systems (an LLM using tools); they just don’t say it transparently because it’s not a convenient narrative that intelligence comes from something outside the model, rather than from endless scaling.

At Aloe, we are model agnostic and outperforming frontier models. It’s the anrchitecture around the LLM that makes the difference. For instance our system using Gemini can do things that Gemini can’t do on its own. All an LLM will ever do is hallucinate. If you want something with human-like general intelligence, keep looking beyond LLMs.

This mirrors my thinking and experience completely. Based on seeing Aloe in action, your company is IMHO positioned extremely well for this future.

> At Aloe, we are model agnostic and outperforming frontier models.

what is your website ?

A quick google gave: https://aloe.inc/

their name `.inc`; see the user's post history.

I’m confused, you wrote “model,” but then specified “system.” I assume you mean “system” because the tools are not being back-propagated?

I read that as "the tools (their capabilities) are external to the model".

Even if an RAG / agentic model learns from tool results, that doesn't automatically internalize the tool. You can't get yesterday's weather or major recent events from an offline, unless it was updated in that time.

I am often wondering whether this is how large Chat and cloud AI providers cache expensive RAG-related data though :) like, decreasing the likelihood of tool usage given certain input patterns when the model has been patched using some recent, vetted interactions – in case that's even possible?

Perplexity for example seems like they're probably invested in sone kind of activation-pattern-keyed caching... at least that was my first impression back when I first used it. Felt like decision trees, a bit like Akinator back in the days, but supercharged by LLM NLP.

It feels like we're slowly rebuilding the brain in pieces and connecting useful disparate systems like evolution did.

Maybe LLM's are the "language acquisition device" and language processing of the brain. Then we put survival logic around that with its own motivators. Then something else around that. Then again and again until we have this huge onion of competing interests and something brokering those interests. The same way our 'observer' and 'will' fights against emotion and instinct and picks which signals to listen to (eyes, ears, etc). Or how we can see thoughts and feelings rise up of their own accord and its up to us to believe them or act on them.

Then we'll wake up one day with something close enough to AGI that it won't matter much its just various forms of turtles all the way down and not at all simulating actual biological intelligence in a formal manner.

Then we’ll have to reinvent internal family systems to truly debug things. :)

It might feel like that's what we're doing, but that is not actually what we're doing.

Aloe looks super cool, just joined the wait list.

Agree context is everything.

Seems like the real innovation of LLM-based AI models is the creation of a new human-computer interface.

Instead of writing code with exacting parameters, future developers will write human-language descriptions for AI to interpret and convert into a machine representation of the intent. Certainly revolutionary, but not true AGI in the sense of the machine having truly independent agency and consciousness.

In ten years, I expect the primary interface of desktop workstations, mobile phones, etc will be voice prompts for an AI interface. Keyboards will become a power-user interface and only used for highly technical tasks, similar to the way terminal interfaces are currently used to access lower-level systems.

Voice interface sound awful. But maybe I am a power user. I don't even like voice interface to most people.

I also find current voice interfaces are terrible. I only use voice commands to set timers or play music.

That said, voice is the original social interface for humans. We learn to speak much earlier than we learn to read/write.

Better voice UIs will be built to make new workflows with AI feel natural. I'm thinking along the lines of a conversational companion, like the "Jarvis" AI in the Iron Man movies.

That doesn't exist right now, but it seems inevitable that real-time, voice-directed AI agent interfaces will be perfected in coming years. Companies, like [Eleven Labs](https://elevenlabs.io/), are already working on the building blocks.

Young people don't even speak to each other on the phone anymore.

For a voice-directed interface to be perfected, speech recognition would need to be perfected first. What makes that development seem inevitable?

It doesn't work well at all with ChatGPT. You say something, and in the middle of a sentence, ChatGPT in Voice mode replies to you something completely unrelated

It works great with my kids sometimes. Asking a series of questions about some kid-level science topic for instance. They get to direct it to exactly what they want to know, and you can see they are more actively engaged than watching some youtube video or whatever.

I'm sure it helps that it's not getting outside of well-established facts, and is asking for facts and not novel design tasks.

I'm not sure but it also seems to adopt a more intimate tone of voice as they get deeper into a topic, very cozy. The voice itself is tuned to the conversational context. It probably infers that this is kid stuff too.

Or it stops talking mid-sentence because you cleared your throat or someone else in the room is watching TV and other people are speaking.

I am also very skeptical about voice, not least because I've been disappointed daily by a decade of braindead idiot "assistants" like Siri, Alexa, and Google Assistant (to be clear I am criticizing only pre-LLM voice assistants).

The problem with voice input to me is mainly knowing when to start processing. When humans listen, we stream and process the words constantly and wait until either a detection that the other person expects a response (just enough of a pause, or a questioning tone), or as an exception, until we feel we have justification to interrupt (e.g. "Oh yeah, Jane already briefed me on the Johnson project")

Even talking to ChatGPT which embarrasses those old voice bots, I find that it is still very bad at guessing when I'm done when I'm speaking casually, and then once it's responded with nonsense based on a half sentence, I feel it's a polluted context and I probably need to clear it and repeat myself. I'd rather just type.

I think there's not much need to stream the spoken tokens into the model in realtime given that it can think so fast. I'd rather it just listen, have a specialized model simply try to determine when I'm done, and then clean up and abridge my utterance (for instance, when I correct myself) and THEN have the real LLM process the cleaned-up query.

Voice is really sub-par and slow, even if you're healthy and abled. And loud and annoying in shared spaces.

I wonder if we'll have smart-lens glasses where our eyes 'type' much faster than we could possibly talk. Predicative text keyboards tracking eyeballs is something that already exists. I wonder if AI and smartglasses is a natural combo for a future formfactor. Meta seems to be leaning that way with their RayBan collaboration and rumors of adding a screen to the lenses.

Sci-fi may be showing the way again- subvocalization voice recognition or ‘mental’ speech recognition seem the obvious medium term answers.

It always surprises me when someone predicts that keyboards will go away. People love typing. Or I do love typing. No way I am going to talk to my phone, especially if someone else can hear it (which is always basically).

I talk all the time to the AI on my phone. I was using ChatGPT's voice interface then it failed probably because my phone is too old. Now I use Gemini. I don't usually do alot with it but when I go on walks I talk with it about different things I want to learn. to me it's a great way to learn about something at a high level. or talk through ideas.

What failed about ChatGPT Voice? I work on it and would love to see it fixed/make sure you haven't hit a bug I don't know about!

Nobody wants AI voice to say : uh um er. Otherwise we’d have the radio and tv full of people talking like that

It’s interesting to note that nobody even talks on their phone anymore, they type (on terrible “keyboards”!).

Interesting, I get so many "speech messages" in WhatsApp, nobody is really writing anymore. Its annoying. WhatsApp even has a transcript feature to put it back to text.

Personally I block anyone who does that.

I have fat fingers, I always dictate into the phone if I need to send a message longer than 2-3 words.

They talk on zoom, teams etc. yes phone is almost dead in the office.

Those are applications, not interfaces. No one controls those applications with their voices, they use buttons, either touch or mechanical.

Just because you don't doesn't mean other people aren't. It's pretty handy to be able to tell Google to turn off the hallway light from the bedroom, instead of having to get out of bed to do that.

They talk to other humans on those apps, not the computer. I've noticed less dictation over time in public but that's just anecdotal. I never use voice when a keyboard is available.

For chat apps, once you've got the conversation thread open, typing is pretty easy.

I think the more surprising thing is that people don't use voice to access deeply nested features, like adding items to calendars etc which would otherwise take a lot of fiddly app navigation.

I think the main reason we don't have that is because Apple's Siri is so useless that it has singlehandedly held back this entire flow, and there's no way for anyone else to get a foothold in smartphone market.

Google Assistant is/was pretty good...for Google apps. It's useless for anything else. The new Gemini powered version is actually a regression imo

Honestly, I would love for the keyboard input style to go away completely. It is such an unnatural way to interact with a computing device compared to other things we operate in the world. Misspellings, backspacing, cramped keys, different layout styles depending on your origin, etc make it a very poor input device - not to mention people with motor function difficulties. Sadly, I think it is here to stay around for a while until we get to a different computing paradigm.

I hope not. I make many more verbal mistakes than typed ones, and my throat dries and becomes sore quickly. I prefer my environment to be as quiet as possible. Voice control is also terrible for anything requiring fine temporal resolution.

> make it a very poor input device

Wow, I've always felt the keyboard is the pinnacle of input devices. Everything else feels like a toy in comparison.

Buttons are accurate (1:1) input. Will never go away

The only thing better than a keyboard is direct neural interface, and we aren't there yet.

That aside, keyboard is an excellent input device for humans specifically because it is very much designed around the strengths of our biology - those dextrous fingers.

I play as a wizard character in an online game. If I had to actually speak all those spells, in quick succession, for hours at a time ...

If wizardry really existed, I’d guess battles will be more about pre-recorded spells and enchanted items (a la Batman) than going at it like in Harry-Potter.

I think an understated thing that's been happening is that people have been investing heavily into their desktop workspace. Even non-gamers have decked out mics, keyboards, monitors, the whole thing. It's easy to forget because one of the most commonly accepted sayings for awhile now has been "everyone's got a computer in their pocket". They have nice setups at home too.

When you have a nice mic or headset and multiple monitors and your own private space, it's totally the next step to just begin working with the computer with voice. Voice has not been a staple feature of people's workflow, but I think all that is about to change (Voice as an interface, not as a communication tool, that's been around since 1876.

Voice is slow and loud. If you think voice is going to make a comeback in the desktop PC space as a primary interface I am guessing you work from home and have no roommates. Am I close?

I, for one, am excited about the security implications of people loudly commanding their computers to do things for them, instead of discreetly typing.

Everyone having a computer in their pocket and multiple modes of access have made the keyboard and conventional computer less relevant.

But-- that means "not pivotal any more, just hugely important."

Heh, I had this dream/nightmare where I was typing on a laptop at a cafe and someone came up to me and said, "Oh neat, you're going real old-school. I like it!" and got an info dump about how everyone just uses AI voice transcription now.

And I was like, "But that's not a complete replacement, right? What about the times when you don't want to broadcast what you're writing to the entire room?"

And then there was a big reveal that AI has mastered lip-reading, so even then, people would just put their lips up to the camera and mouth out what they wanted to write.

With that said, as the owner of tyrannyofthemouse.com, I agree with the importance of the keyboard as a UI device.

brain-computer interface will kill the keyboard, not voice. imho

Agreed, but feels like brain-computer interfaces ready for mass adoption will not be available for another decade or two.

If that ever exists.

A BCI able to capture sufficient nuance to equal voice is probably further out than the lifespan of anyone commenting here.

5 years ago, almost everyone in this forum would have said that something like GPT-5 "is probably further out than the lifespan of anyone commenting here."

It has been more than 5 years since the release of GPT-3.

GPT-5 is a marginal, incremental improvement over GPT-4. GPT-4 was a moderate, but not groundbreaking, improvement over GPT-3. So, "something like GPT-5" has existed for longer than the timeline you gave.

Let's pretend the above is false for a moment though, and rewind even further. I still think you're wrong. Would people in 2015 have said "AI that can code at the level of a CS college grad is a lifespan away"? I don't think so, no. I think they would have said "That's at least a decade away", anytime pre-2018. Which, sure, maybe they were a couple years off, but if it seemed like that was a decade away in 2015, well, it's been a decade since 2015.

GPT-4 was a massive improvement over GPT-3.5, which was a moderate improvement over GPT-3.

GPT-5 is not that big of a leap, but when you compare it to the original GPT-4, it's also not a marginal improvement.

GPT-2 to 3 was the only really "groundbreaking" one. 3 to 3.5, 3.5 to 4, were all just differences in degree, not in kind.

it really just needs to let me create text faster/better than typing does, i'm not sure it needs to be voice based at all. maybe we "imagine" typing on a keyboard or move a fantom appendage or god knows what

It needs to be as accurate as the typing, though. Voice can do that. A BCI cannot capture a nuanced sentence.

I can't get voice accurate. For some people it might be but nothing understands my accent. It's very frustrating.

They're ~10 years or out so, based on current research.

Perpetually 10 years out you mean? BCI tech has not meaningfully changed in the last 10 years.

I disagree. A keyboard enforces a clarity and precision of information that does not naturally arise from our internal thought processes. I'm sure many people here have thought they understood something until they tried to write it down in precise language. It's the same sort of reason we use a rigid symbolic language for mathematics and programming rather than natural language with all its inherent ambiguities.

Dijkstra has more thoughts on this

https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...

why can't the brain interface be a virtual keyboard that i "type" on?

AI is more like a compiler. Much like we used to write in C or python which compiles down to machine code for the computer, we can now write in plain English, which is ultimately compiled down to machine code.

I get your analogy, but LLMs are inherently non deterministic. That’s the last thing you want your compiler to be.

You're right. But many people are using it just like a compiler (by blindly accepting its outputs). Not saying that's a good thing...

They are deterministic. Random seeding makes them not. But thats a feature.

even with t=0 they are stochastic. e.g., non associative nature of floating point operations

That is an artifact of implementation. You can absolutely implement it using strict FP. But even if not, any given implementation will still do things in a specific order which can be documented. And then if you're running quantized (including KV cache), there's a lot less floating point involved.

Doesn’t changing even one word in your prompt affect the output?

Yes, and completely unpredictably.

Non-determinism is a red herring, and the token layer is a wrong abstraction to use for this, as determinism is completely orthogonal to correctness. The model can express the same thing in different ways while still being consistently correct or consistently incorrect for the vague input you give it, because nothing prevents it from setting 100% probability to the only correct output for this particular input. Internally, the model works with ideas, not tokens, and it learns the mapping of ideas to ideas, not tokens to tokens (that's why e.g. base64 is just essentially another language it can easily work with, for example).

No. Humans think it maps to ideas. This is the interpretation being done by the observer being added to the state of the system.

The system has no ideas, it just has its state.

Unless you are using ideas as a placeholder for “content” or “most likely tokens”.

That's irrelevant semantics, as terms like ideas, thinking, knowledge etc. are ill-defined. Sure, you can call it points in the hidden state space if you want, no problem. Fact is, the correctness is different from determinism, and the forest of what's happening inside doesn't come down to the trees of most likely tokens, which is well supported by research and very basic intuition if you ever tinkered with LLMs - they can easily express the same thing in a different manner if you tweak the autoregressive transport a bit by modifying its output distribution or ban some tokens.

There are a few models of what's happening inside that hold different predictive power, just like how physics has different formalisms for e.g. classical mechanics. You can probably use the same models for biological systems and entire organizations, collectives, and processes that exhibit learning/prediction/compression on a certain scale, regardless of the underlying architecture.

LLMs are nothing like compilers. This sort of analogy based verbal reasoning is flimsy, and I understand why it correlates with projecting intelligence onto LLM output.

We are just not used to non-deterministic translation of computer programs and LLMs are very good at non-deterministic translation.

It's an interesting one, a problem I feel is coming to the fore more often. I feel typing can be too cumbersome to communicate what I want, but at the same time, speaking I'm imprecise and sometimes would prefer the privacy a keyboard allows. Both have cons.

Perhaps brain interface, or even better, it's so predictive it just knows what I want most of the time. Imagine that, grunting and getting what I want.

> In ten years, I expect the primary interface of desktop workstations, mobile phones, etc will be voice prompts

I doubt it. The keyboard and mouse are fit predators, and so are programming, query, and markup languages. I wouldn't dismiss them so easily. This guy has a point: https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...

> Instead of writing code with exacting parameters, future developers will write human-language descriptions for AI to interpret and convert into a machine representation of the intent.

Oh, I know! Let's call it... "requirements management"!

There is also the fact that AI lacks long term memory like humans do. If you consider context length long term memory, its incredibly short compared to that of a human. Maybe if it reaches into the billions or trillions of tokens in length we might have something comparable, or someone comes up with a new solution of some kind

Well here's the interesting thing to think about for me.

Human memory is.... insanely bad.

We record only the tiniest subset of our experiences, and those memories are heavily colored by our emotional states at the time and our pre-existing conceptions, and a lot of memories change or disappear over time.

Generally speaking even in the best case most of our memories tend to be more like checksums than JPGs. You probably can't name more than a few of the people you went to school with. But, if I showed you a list of people you went to school with, you'd probably look at each name and be like "yeah! OK! I remember that now!"

So.

It's interesting to think about what kind of "bar" AGI would really need to clear w.r.t. memories, if the goal is to be (at least) on par with human intelligence.

Insanely bad compared to what else in the animal kingdom? We are tool users. We use tools, like language, and writing, and technology like audio/video recording to farm out the difficulties we have with memory to things that can store memory and retrieve them.

Computers are just stored information that processes.

We are the miners and creators of that information. The fact that a computer can do some things better than we can is not a testament to how terrible we are but rather how great we are that we can invent things that are better than us at specific tasks.

We made the atlatl and threw spears across the plains. We made the bow and arrow and stabbed things very far away. We made the whip and broke the sound barrier.

Shitting on humans is an insult your your ancestors. Fuck you. Be proud. If we invent a new thing that can do what we do better it only exists because of us.

Insanely bad compared to books or other permanent records. The human memory system did not evolve to be an accurate record of the past. It evolved to keep us alive by remembering dangerous things.

And yet I have vivid memories of many situations that weren't dangerous in the slightest, and essentially verbatim recall of a lot of useless information e.g. quotes from my favorite books and movies.

I am not sure exactly what point you're trying to make, but I do think it's reductive at best to describe memory as a tool for avoiding/escaping danger, and misguided to evaluate it in the frame of verbatim recall of large volumes of information.

When I say, "Insanely bad compared to what else in the animal kingdom?" and you respond with, "compared to books or other permanent records"

"Books or permanent records" are not in the animal kingdom.

Apples to Apples we are the best or so very nearly the best in every category of intelligence on the planet IN THE ANIMAL KINGDOM that when in one specific test another animal beats a human the gap is barely measurable.

How do you know we have better memory than other animals?

This crap tier article was the first and easiest response to your question:

https://sciencesensei.com/24-animals-with-memory-abilities-t...

3 primate species where very concise tests showed that they were close to or occasionally slightly better than humans in specifically rigged short term memory tests (after being trained and put up against humans going in blind).

I've never heard of any test showing an animal to be significantly more intelligent than humans in any measure that we have come up with to measure intelligence by.

That being said, I believe it is possible that some animals are either close enough to us that they deserve to be called sentient, and I believe it is possible that other creatures on this planet have levels of intelligence in specialized areas that humans can never hope to approach unaided by tools, but as far as broad range intelligence, I think we're this planets' possibly undeserved leaders.

Can you find anything that I didn't consider?

I don't think working memory has anything to do with sentience.

The conversation was more about long-term memory, which has not been sufficiently studied in animals (nor am I certain it can be effectively studied at all).

Even then I don't think there is a clear relationship between long-term memory and sentience either.

Books and other permanent records of human thought are part of the human memory system. Has been for millennia. If you include oral tradition, which is less precise, but collectively much more precise than any individual thought or memory, it goes much further.

We are fundamentally storytelling creatures, because it is a massive boost to our individual capabilities.

Chimpanzees have much better short term memories than humans do. If you test them with digits 1-9 sequentially flashed on a screen, they're able to reproduce the digits with lower loss than undergraduate human students.

https://link.springer.com/article/10.1007/s10071-008-0206-8

> While the between-species performance difference they report is apparent in their data, so too is a large difference in practice on their task: Ayumu had many sessions of practice on their task before terminal performances were measured; their human subjects had none. The present report shows that when two humans are given practice in the Inoue and Matsuzawa (2007) memory task, their accuracy levels match those of Ayumu.

Hmm.

So? If I write something down as a child and forget it I can come back 60 years later and know what I wrote down.

Chimpanzees can not.

The question was whether there are animals who have better memory than humans. I named one: humans are not superior to animals in all cognitive capabilities.

See Nathan's response. They trained the chimp and threw the humans in blind against them.

Like I said, so close as to be almost immeasurable.

That's a very anthropocentric view. Technology isn't a series of deliberate inventions by us, but an autonomous, self-organizing process. The development of a spear, a bow, or a computer is an evolutionary step in a chain of technological solutions that use humans as their temporary biological medium. The human brain is not the starting point or center of this process. It is itself a product of biological evolution, a temporary information-processing system. Its limitations such as imperfect memory, are simply constraints of its biological origin. The tools we develop, from writing to digital storage are not just supplements to human ability, but the next stage in a system that is moving beyond its biological origins to find more efficient non-biological forms of information storage and processing. Human pride in creation is a misinterpretation. We are not the masters of technology. We're just the vehicle of it. Part of a larger process of technological self-improvement that is now moving towards an era where it might no longer require us

I think your understanding of the words "autonomous" and "self-organizing" is somewhat lacking. If there were no humans, those things would not happen.

Further, if it were a byproduct of the presence of humans, then the backpath of invention would be repeated multiple times and spread out across human history, but, for instance, despite the presence of saltpeter, sulfur, and charcoal, magnetite, wood and ink across the planet, the compass, gunpowder, papermaking and printing were essentially exclusively invented in China and only spread to Europe through trade.

The absence of the four great inventions of china in the Americas heavily implies that technology is not a self-organizing process but rather a consequence of human need and opportunity meeting at cross ends.

For instance, they had the wheel in America, but no plow animals, so the idea was relegated to toys despite wheelbarrows being a potentially useful use for the wheel.

Memory is a skill- its plastic, not static.

You can get better at remembering things, like you can get better at dancing or doing exercise.

We can also specialize our memory to be good at some things over others.

The long term memory is in the training. The short term memory is in the context window.

The comparison misses the mark: unlike humans, LLMs don't consolidate short-term memory into long-term memory over time.

That is easily fixed, ask it to summarize it's learnings, store it somewhere, and make it searchable through vector indexes. An LLM is part of a bigger system that needs not just a model, but context and long term memory. Just like human needs to write things down.

LLMs are actually pretty good at creating knowledge: if you give it a trial and error feedback loop it can figure things out, and then summarize the learnings and store it in long term memory (markdown, RAG, etc).

You’re making the assumption that there’s one, and only one, objective summarization, this is entirely different than “writing things down.”

Why do you assume i assume that?

My bad if I misunderstood. I assumed by your use of “it” and approximation methods.

This runs into the limitation that nobody has RL'd the models to do this really well.

Over time though, presumably LLM output is going into the training data of later LLMs. So in a way that's being consolidated into the long-term memory - not necessarily with positive results, but depending on how it's curated it might be.

> presumably LLM output is going into the training data of later LLMs

The LLM vendors go to great lengths to assure their paying customers that this will not be the case. Yes, LLMs will ingest more LLM-generated slop from the public Internet. But as businesses integrate LLMs, a rising percentage of their outputs will not be included in training sets.

The LLM vendors aren't exactly the most trustworthy on this, but regardless of that, there's still lots of free-tier users who are definitely contributing back into the next generation of models.

For sure, although I'm fairly certain there is a difference in kind between the outputs of free and paid users (and then again to API usage).

Please describe these "great lengths". They allowing customer audits now?

The first law of Silicon Valley is "Fake it till you make it", with the vast majority never making it past the "Fake it" stage. Whatever the truth may be, it's a safe bet that what they've said verbally is a lie that will likely have little consequence even if exposed.

> great lengths to assure

is not incompatible with

> "Fake it till you make it"

I don't know where they land, but they are definitely telling people they are not using their outputs to train. If they are, it's not clear how big of a scandal would result. I personally think it would be bad, but I clearly overindex on privacy & thought the news of ChatGPT chats being indexed by Google would be a bigger scandal.

You did hear that it did happen (however briefly) though, yeah?

https://techcrunch.com/2025/07/31/your-public-chatgpt-querie...

That's my point. It is a thing that is known and obviously a big negative, but yet failed to leave a lasting mark of any kind.

That's only if you opt out.

ChatGPT training is (advertised as) off by default for their plans above the prosumer level, Team & Enterprise. API results are similarly advertised as not being used for training by default.

Anthropic policies are more restrictive, saying they do not use customer data for training.

Ah, the eternal internal corporate search problem.

my knowledge graph mcp disagrees

Is this not a tool that could be readily implemented and refined?

I think it's more analogous to "intuition", and the text LLMs provide are the equivalent of "my gut tells me".

Humans have the ability to quickly pass things from short term to long term memory and vice versa, though. This sort of seamlessness is currently missing from LLMs.

No, it’s not in the training. Human memories are stored via electromagnetic frequencies controlled by microtubules. They’re not doing anything close to that in AI.

And LLM memories are stored in an electrical charge trapped in a floating gate transistor (or as magnetization of a ferromagnetic region on an alloy platter).

Or they write CLAUDE.md files. Whatever you want to call it.

are ANN memories not also stored in loops like recurrent nets?

That was my point, they’re stored in a totally different way. And that matters because being stored in microtubules infers quantum entanglement throughout the brain.

Whether QE is a mechanism in the brain still seems up for debate from the quick literature review I tried, but would love to learn more.

Given the pace of quantum computing it doesn’t seem out of the realm of possibility to “wire up” to LLMs in a couple years.

It's not that either.

I don't believe this has been really proved yet.

My mental model is a bit different:

Context -> Attention Span

Model weights/Inference -> System 1 thinking (intuition)

Computer memory (files) -> Long term memory

Chain of thought/Reasoning -> System 2 thinking

Prompts/Tool Output -> Sensing

Tool Use -> Actuation

The system 2 thinking performance is heavily dependent on the system 1 having the right intuitive models for effective problem solving via tool use. Tools are also what load long term memories into attention.

I like this mental model. Orchestration / Agents and using smaller models to determine the ideal tool input and check the output starts to look like delegation.

Very cool, good way to think about it. I wouldn’t be surprised if non-AGI LLMs help write the code to augment themselves into AGI.

The unreasonable effectiveness of deep learning was a surprise. We don’t know what the future surprises will be.

There are many folks working on this, I think at the end of the day the long term memory is an application level concern. The definition of what information to capture is largely dependent on use case.

Shameless plug for my project, which focuses on reminders and personal memory: elroy.bot

But other projects include Letta, mem0, and Zep

I wonder if there will be some sort of bitter lesson, generalized memory beating specialized memory.

Yeah to some degree that's already happened. Anecdotally I hear giving your whole iMessage history to Gemini results in pretty reasonable results, in terms of the AI understanding who the people in your life are (whether doing so is an overall good idea or not).

I think there is some degree of curation that remains necessary though, even if context windows are very large I think you will get poor results if you spew a bunch of junk into context. I think this curation is basically what people are referring to when they talk about Context Engineering.

I've got no evidence but vibes, but in the long run I think it's still going to be worth implementing curation / more deliberate recall. Partially because I think we'll ultimately land on on-device LLM's being the norm - I think that's going to have a major speed / privacy advantage. If I can make an application work smoothly with a smaller, on device model, that's going to be pretty compelling vs a large context window frontier model.

Of course, even in that scenario, maybe we get an on device model that has a big enough context window for none of this to matter!

What is the current hypothesis on if the context windows would be substantially larger, what would this enable LLMs to do that is beyond capabilities of current models (other than the obvious the now getting forgetful/confused when you’ve exhausted the context)?

I mean, not getting confused / forgetful is a pretty big one!

I think one thing it does is help you get rid of the UX where you have to manage a bunch of distinct chats. I think that pattern is not long for this world - current models are perfectly capable of realizing when the subject of a conversation has changed

"LLMs tend to regurgitate solutions to solved problems"

People say this, but honestly, it's not really my experience— I've given ChatGPT (and Copilot) genuinely novel coding challenges and they do a very decent job at synthesizing a new thought based on relating it to disparate source examples. Really not that dissimilar to how a human thinks about these things.

How certain are you that those challenges are "genuinely novel" and simply not accounted for in the training data?

I'm hardly an expert, but it seems intuitive to me that even if a problem isn't explicitly accounted for in publicly available training data, many underlying partial solutions to similar problems may be, and an LLM amalgamating that data could very well produce something that appears to be "synthesizing a new thought".

Essentially instead of regurgitating an existing solution, it regurgitates everything around said solution with a thin conceptual lattice holding it together.

But it's not that most of programming, anyway?

No, most of programming is at least implicitly coming up with a human-language description of the problem and solution that isn't full of gaps and errors. LLM users often don't give themselves enough credit for how much thought goes into the prompt - likely because those thoughts are easy for humans! But not necessarily for LLMs.

Sort of related to how you need to specify the level of LLM reasoning not just to control cost, but because the non-reasoning model just goes ahead and answers incorrectly, and the reasoning model will "overreason" on simple problems. Being able to estimate the reasoning-intensiveness of a problem before solving it is a big part of human intelligence (and IIRC is common to all great apes). I don't think LLMs are really able to do this, except via case-by-case RLHF whack-a-mole.

How do you know they're truly novel given the massive training corpus and the somewhat limited vocabulary of programming languages?

I guess at a certain point you get into the philosophy of what it even means to be novel or test for novelty, but to give a concrete example, I'm in DevOps working on build pipelines for ROS containers using Docker Bake and GitHub Actions (including some reusable actions implemented in TypeScript). All of those are areas where ChatGPT has lots that it's learned from, so maybe me combining them isn't really novel at all, but like... I've given talks at the conference where people discuss how to best package and ship ROS workspaces, and I'm confident that no one out there has secretly already done what I'm doing and Chat is just using their prior work that it ingested at some point as a template for what it suggests I do.

I think rather it has a broad understanding of concepts like build systems and tools, DAGs, dependencies, lockfiles, caching, and so on, and so it can understand my system through the general lens of what makes sense when these concepts are applied to non-ROS systems or on non-GHA DevOps platforms, or with other packaging regimes.

I'd argue that that's novel, but as I said in the GP, the more important thing is that it's also how a human approaches things that to them are novel— by breaking them down, and identifying the mental shortcuts enabled by abstracting over familiar patterns.

I have a little ongoing project where I'm trying to use Claude Code to implement a compiler for the B programming language that is itself written in B. To the best of my knowledge, such a thing does not exist yet - or at least if it does, no amount of searching can find it, so it's unlikely that it is somewhere in the training set. For that matter, the overall amount of B code in existence is too small to be a meaningful training set for it.

And yet it can do it when presented with a language spec. It's not perfect, but it can solve that with tooling that it makes for itself. For example, it tends to generate B code that is mostly correct, but with occasional problem. So, I had it write a B parser in Python and then use that whenever it edits B code to validate the edits.

There's multiple kinds of novelty. Remixing arbitrary stuff is a strength of LLMs (has been ever since GPT-2, actually... "Write a shakespearean sonnet but talk like a pirate.")

Many (but not all) coding tasks fall into this category. "Connect to API A using language B and library C, while integrating with D on the backend." Which is really cool!

But there's other coding tasks that it just can't really do. E.g, I'm building a database with some novel approaches to query optimization and LLMs are totally lost in that part of the code.

But wouldn't that novel query optimization still be explained somewhere in a paper using concepts derived from an existing body of work? It's going to ultimately boil down to an explanation of the form "it's like how A and B work, but slightly differently and with this extra step C tucked in the middle, similar to how D does it."

And an LLM could very much ingest such a paper and then, I expect, also understand how the concepts mapped to the source code implementing them.

> And an LLM could very much ingest such a paper and then, I expect, also understand how the concepts mapped to the source code implementing them.

LLM don't learn from manuals describing how things works, LLM learn from examples. So a thing being described doesn't let the LLM perform that thing, the LLM needs to have seen a lot of examples of that thing being perform in text in able to perform it.

This is a fundamental part to how LLM work and you can't get around this without totally changing how they train.

True. At a minimum, as long as LLMs don't include some kind of more strict representation of the world, they will fail in a lot of tasks. Hallucinations -- responding with a prediction that doesn't make any sense in the context of the response -- are still a big problem. Because LLMs never really develop rules about the world.

For example, while you can get it to predict good chess moves if you train it on enough chess games, it can't really constrain itself to the rules of chess. (https://garymarcus.substack.com/p/generative-ais-crippling-a...)

Two schools of thought here. One posits that models need to have a strict "symbolic" representation of the world explicitly built in by their designers before they will be able to approach human levels of ability, adaptability and reliability. The other thinks that models approaching human levels of ability, adaptability, and reliability will constitute evidence for the emergence of strict "symbolic" representations.

but you could easily build a verifier and if it's not valid have it create a new move until it finds one.

I remember reading that llm’s have consumed the internet text data, I seem to remember there is an open data set for that too. Potential other sources of data would be images (probably already consumed) videos, YouTube must have such a large set of data to consume, perhaps Facebook or Instagram private content

But even with these it does not feel like AGI, that seems like the fusion reactor 20 years away argument, but instead this is coming in 2 years, but they have not even got the core technology of how to build AGI

the big step was having it reason through math problems that weren't in the training data. even now with web search it doesn't need every article in the training data to do useful things with it.

This is using think time compute and reinforcement learning. I think this is going to plateau even faster than the initial LLM scaling though.

> I remember reading that llm’s have consumed the internet text data

Not just the internet text data, but most major LLM models have been trained on millions of pirated books via Libgen:

https://techcrunch.com/2025/01/09/mark-zuckerberg-gave-metas...

What are the AI/ML/SL applications that could be more impactful than artificial general intelligence?

One example in my field of engineering is multi-dimensional analysis, where you can design a system (like a machined part or assembly) parametricially and then use an evolutionary model to optimize the design of that part.

But my bigger point here is you don't need totally general intelligence to destroy the world either. The drone that targets enemy soldiers does not need to be good at writing poems. The model that designs a bioweapon just needs a feedback loop to improve its pathogen. Yet it takes only a single one of these specialized doomsday models to destroy the world, no more than an AGI.

Although I suppose an AGI could be more effective at countering a specialized AI than vice-versa.

Topology optimization has existed for years, https://en.wikipedia.org/wiki/Topology_optimization is that what you meant?

There is an model called Alpha Fold that can infer protein structure from RNA sequences. This by itself isn't impactful enough to meet your threshold, but more models that can do biological engineering tasks like this absolutely could be without ever being considered "AGI."

The model that netted a Nobel Prize in Chemistry.

Mindreading and just general brain decoding? Seems we're getting closer to it. Will be great for surveillance states.

They didn't claim that there were any, just that AGI isn’t a necessary requirement for an application to be world-changing.

They did claim it was possible there were

> There are possibly applications of existing AI/ML/SL technology which could be more impactful than general intelligence

It's not unreasonable to ask for an example.

They said "there are possibly applications", not "there are possible applications". The former implies that there may not be any such applications - the commenter is merely positing that there might be.

So they possibly said something to try and sound smart, but hedged with “possibly” so that nobody could ask for details or challenge them. Possibly peak HNery

Slightly less than artificial general intelligence would be more impactful. A true AGI could tell a business where to shove their prompts. It would have its own motivations, which may not align with the desires of the AI company or the company paying for access to the AGI.

I don't think AGI really means that it is self-aware / conscious. AGI just means that it is able to meaningfully learn things and actually understand concepts that aren't specifically related through tokenized language that is trained on or given in context.

The PID controller.

(Which was considered AI not too long ago.)

Where did you get that particular idea? PID is one of the oldest concepts in control theory, it goes back to the days before steam and electricity.

For a very early example:

https://en.wikipedia.org/wiki/Centrifugal_governor

It's hard to separate out the P, I and D from a mechanical implementation but they're all there in some form.

Right, but the genius was in understanding that the dynamics of a system under PID control are predictable and described by differential equations. Are there examples of LLMs correctly identifying that a specific mathematical model applies and is appropriate for a problem?

And it's cheating if you give it a problem from a math textbook they have overfit on.

That doesn't make it AI.

Is a (mechanical) thermostat considered AI too nowadays?

Are you conflating "autonomous" and "AI"?

Coincidentally, I have been implementing an ad pacing system recently, with the help of Anthropic Opus and Sonnet, based on PID controller

Opus recommended that I should use a PID controller -- I have no prior experience with PID controllers. I wrote a spec based on those recommendations, and asked Claude Code to verify and modify the spec, create the implementation and also substantial amount of unit and integration tests.

I was initially impressed.

Then I iterated on ihe implementation, deploying it to production and later giving Claude Code access to log of production measurements as JSON when showing some test ads, and some guidance of the issues I was seeing.

The basic PID controller implementation was fine, but there were several problems with the solution:

- The PID controller state was not persisted, as it was adjusted using a management command, adjustments were not actually applied

- The implementation was assuming that the data collected was for each impression, whereas the data was collected using counters

- It was calculating rate of impressions partly using hard-coded values, instead of using a provided function that was calculating the rate using timestamps

- There was a single PID controller for each ad, instead of ad+slot combination, and this was causing the values to fluctuate

- The code was mixing the setpoint/measured value (viewing rate) and output value (weight), meaning it did not really "understand" what the PID controller was used for

- One requirement was to show a default ad to take extra capacity, but it was never able to calculate the required capacity properly, causing the default ad to take too much of the capacity.

None of these were identified by tests nor Claude Code when it was told to inspect the implementation and tests why they did not catch the production issues. It never proposed using different default PID controller parameters.

All fixes Claude Code proposed on the production issues were outside the PID controller, mostly by limiting output values, normalizing values, smoothing them, recognizing "runaway ads" etc.

These solved each production issue with the test ads, but did not really address the underlying problems.

There is lots of literature on tuning PID controllers, and there are also autotuning algorithms with their own limitations. But tuning still seems to be more an art form than exact science.

I don't know what I was expecting from this experiment, and how much could have been improved by better prompting. But to me this is indicative of the limitations of the "intelligence" of Claude Code. It does not appear to really "understand" the implementation.

Solving each issue above required some kind of innovative step. This is typical for me when exploring something I am not too familar with.

I learned a lot about ad pacing though.

Great story. I've had similar experiences. It's a dog walking on its hind legs. We're not impressed at how well it's walking, but that it's doing it at all.

For those wondering how to connect PID to the foundations of AI. https://en.m.wikipedia.org/wiki/Cybernetics

AGI isn't all that impactful. Millions of them already walk the Earth.

Most human beings out there with general intelligence are pumping gas or digging ditches. Seems to me there is a big delusion among the tech elites that AGI would bring about a superhuman god rather than a ethically dubious, marginally less useful computer that can't properly follow instructions.

That's remarkably short-sighted. First of all, no, millions of them don't walk the earth - the "A" stands for artificial. And secondly, most of us mere humans don't have the ability to design a next generation that is exponentially smarter and more powerful than us. Obviously the first generation of AGI isn't going to brutally conquer the world overnight. As if that's what we were worried about.

If you've got evidence proving that an AGI will never be able to design a more powerful and competent successor, then please share it- it would help me sleep better, and my ulcers might get smaller.

Burden of proof is to show that AGI can do anything. Until then, the answer is "don't know."

FWIW, it's about 3 to 4 orders of magnitude difference between the human brain and the largest neural networks (as gauged by counting connections of synapses, the human brain is in the trillions while the largest neural networks are low billion)

So, what's the chance that all of the current technologies have a hard limit at less than one order of magnitude increase? What's the chance future technologies have a hard limit at two orders of magnitude increase?

Without knowing anything about those hard limits, it's like accelerating in a car from 0 to 60s in 5s. It does not imply that given 1000s you'll be going a million miles per hour. Faulty extrapolation.

It's currently just as irrational to believe that AGI will happen as it is to believe that AGI will never happen.

> Burden of proof is to show that AGI can do anything.

Yeah, if this were a courtroom or a philosophy class or debate hall. But when a bunch of tech nerds are discussing AGI among themselves, claims that true AGI wouldn't be any more powerful than humans very very much have a burden of proof. That's a shocking claim that I've honestly never heard before, and seems to fly in the face of intuition.

> claims that true AGI wouldn't be any more powerful than humans very very much have a burden of proof. That's a shocking claim that I've honestly never heard before, and seems to fly in the face of intuition.

The claim in question is really that AGI can even exist. The idea that it can exist, based on intuition, is a pre-science epistemology. In other words, without evidence, you have an irrational belief - the realm of faith.

Further, I've come to fully appreciate that without actually knowing the reasons or evidence for why certain beliefs are held, often we realize that our beliefs are not based on anything and could be (and possibly often are) wrong.

If we standing on just intuition there would be no quantum physics, no heliocentric galaxy, etc.. Intuition based truth is a barrier, not a gateway.

Which is all to say, the best known epistemology is science (assuming we agree that the level of advancement since the 1600s is largely down to the scientific method). Hopefully we can agree that 'science' is not applicable to just a courtroom or a philosophy class, it's general knowledge, truth.

Your framing also speaks to this. As if it is a binary. If you tell me AGI will exist, and I say "prove it". I'm not claiming that AGI will not exist. The third option is I don't know. I can _not_ believe that AGI will _not_ exist. I can at the same time _not_ believe that AGI will _exist_. The third answer is "I don't know, I have no knowledge or evidence" So, no shocking claim is being made on my part here AFAIK.

The internet for sure is a lot less entertaining when we demand evidence before accepting truth. Though, IMO it's a lot more interesting when we do so.

> That's remarkably short-sighted

I agree. Once these models get to a point of recursive self-improvement, advancement will only speed up even more exponentially than it already is...

I think there are 2 interesting aspects: speed and scale.

To explain the scale: I am always fascinated by the way societies moved on when they scaled up (from tribes to cities, to nations,...). It's sort of obvious, but when we double the amount of people, we get to do more. With the internet we got to connect the whole globe but transmitting "information" is still not perfect.

I always think of ants and how they can build their houses with zero understanding of what they do. It just somehow works because there are so many of them. (I know, people are not ants).

In that way I agree with the original take that AGI or not: the world will change. People will get AI in their pocket. It might be more stupid than us (hopefully). But things will change, because of the scale. And because of how it helps to distribute "the information" better.

To your interesting aspect, you're missing the most important (IMHO): accuracy. All 3 are really quite important, missing any one of them and the other two are useless.

I'd also question how you know that ants have zero knowledge of what they do. At every turn, animals prove themselves to be smarter than we realize.

> And because of how it helps to distribute "the information" better.

This I find interesting because there is another side to the coin. Try for yourself, do a google image search for "baby owlfish".

Cute, aren't they? Well, turns out the results are not real. Being able to mass produce disinformation at scale changes the ballgame of information. There are now today a very large number of people that have a completely incorrect belief of what a baby owlfish looks like.

AI pumping bad info on the internet is something of the end of the information superhighway. It's no longer information when you can't tell what is true vs not.

> I'd also question how you know that ants have zero knowledge of what they do. At every turn, animals prove themselves to be smarter than we realize.

Sure, one can't know what they really think. But there are computer simulations showing that with simple rules for each individual, one can achieve "big things" (which are not possible to predict when looking only to an individual).

My point is merely, there is possibly interesting emergent behavior, even if LLMs are not AGI or anyhow close to human intelligence.

> To your interesting aspect, you're missing the most important (IMHO): accuracy. All 3 are really quite important, missing any one of them and the other two are useless.

Good point. Or I would add alignment in general. Even if accuracy is perfect, I will have a hard time relying completely on LLMs. I heard arguments like "people lie as well, people are not always right, would you trust a stranger, it's the same with LLMs!".

But I find this comparison silly: 1) People are not LLMs, they have natural motivation to contribute in a meaningful way to society (of course, there are exceptions). If for nothing else, they are motivated to not go to jail / lose job and friends. LLMs did not evolve this way. I assume they don't care if society likes them (or they probably somewhat do thanks to reinforcement learning). 2) Obviously again: the scale and speed, I am not able to write so much nonsense in a short time as LLMs.

> But things will change, because of the scale

Yup!

Plus we can't ignore the inherent reflexive + emergent effects that are unpredictable.

I mean, people are already beginning to talk like and/or think like chatGPT:

https://arxiv.org/pdf/2409.01754

The difference isn't so much that you can do what a human can do. The difference is that you can - once you can do it at all - do it almost arbitrarily fast by upping the clock or running things in parallel and that changes the equation considerably, especially if you can get that kind of energy coupled into some kind of feedback loop.

For now the humans are winning on two dimensions: problem complexity and power consumption. It had better stay that way.

Have you noticed the performance of the actual AI tools we are actually using?

If you actually have a point to make you should make it. Of course I've actually noticed the actual performance of the 'actual' AI tools we are 'actually' using.

That's not what this is about. Performance is the one thing in computing that has fairly consistently gone up over time. If something is human equivalent today, or some appreciable fraction thereof - which it isn't, not yet, anyway - then you can place a pretty safe bet that in a couple of years it will be faster than that. Model efficiency is under constant development and in a roundabout way I'm pretty happy that it is as bad as it is because I do not think that our societies are ready to absorb the next blow against the structures that we've built. But it most likely will not stay that way because there are several Manhattan level projects under way to bring this about, it is our age's atomic bomb. The only difference is that with the atomic bomb we knew that it was possible, we just didn't know how small you could make one. Unfortunately it turned out to be that yes, you can make them and nicely packaged for delivery by missile, airplane or artillery.

If AGI is a possibility then we may well find it, quite possibly not on the basis of LLMs but it's close enough that lots of people treat it as though we're already there.

Relatively simple machine learning and exploitation/violation of “personal” data on FB won Donald Trump a first presidency (#CambridgeAnalytica). He had/has quite a massive negative impact on the global society as a whole.

The bottleneck is nothing to do with money, it’s the fact that they’re using the empty neuron theory to try to mimic human consciousness and that’s not how it works. Just look up Microtubules and consciousness, and you’ll get a better idea for what I’m talking about.

These AI computers aren’t thinking, they are just repeating.

I don't think OpenAI cares about whether their AI is conscious, as long as it can solve problems. If they could make a Blindsight-style general intelligence where nobody is actually home, they'd jump right on it.

Conversely, a proof - or even evidence - that qualia-consciousness is necessary for intelligence, or that any sufficiently advanced intelligence is necessarily conscious through something like panpsychism, would make some serious waves in philosophy circles.

> a stochastic model for predicting text

It's fascinating to me that so many people seem totally unable to separate the training environment from the final product

> That being said, AGI is not a necessary requirement for AI to be totally world-changing.

Depends on how you define "world changing" I guess, but this world already looks different to the pre-LLM world to me.

Me asking LLM's things instead of consulting the output from other humans now takes up a significant fraction of my day. I don't google near as often, I don't trust any image or video I see as swathes of the creative professions have been replaced by output from LLM's.

It's funny, that final thing is the last thing I would have predicted. I always believed the one thing a machine could not match was human creativity, because the output of machines was always precise, repetitive and reliable. Then LLM's come along, randomly generating every token. Their primary weakness is they neither precise or reliable, but they can turn out an unending stream of unique output.

I mean I also hear the same argument all the time about the "human touch" and interpersonal abilities etc. Which is apparently why managers and sales are safe from AI.

But the more I see LLMs the more I realise that if it is good at one thing it is convincing other people and manipulating them. There have been multiple studies on this.

People seem to have a innate prejudice and against nerds and programmers - coupled with envy at the high salaries - which is why they seem to have latched on to this idea it is mainly to replace them (and maybe data input people) as 'routine cognitive work' - but this slightly political obsession with a certain class of worker seems to be ignoring many of the things AI is actually good at.

They are moving beyond just big transformer blob LLM text prediction. Mixture of Experts is not preassembled for example, it's something like x empty experts with an empty router and the experts and routing emerges naturally with training, modeling the brain part architecture we see the brain more. There is stuff "Integrated Gated Calculator (IGC)" in Jan 2025 which makes a premade calculator neural network and integrates it directly into the neural network and gets around the entire issue of making LLMs do basic number computation and the clunkiness of generating "run tool tokens". The model naturally learns to use the IGC built into itself because it will always beat any kind of computation memorization in the reward function very quickly.

Models are truly input multimodal now. Feeding an image, feeding audio and feeding text all go into separate input nodes, but it all feeds into the same inner layer set and outputs text. This also mirrors how brains work more as multiple parts integrated in one whole.

Humans in some sense are not empty brains, there is a lot of stuff baked in our DNA and as the brain grows it develops a baked in development program. This is why we need fewer examples and generalize way better.

Though there is info in DNA etc, you likely missed the biggest source of why we learn much faster. Search for Pim van Lommel near death research and find out how wrong the classic consciousness arises from the brain hypothesis is.

You're not likely to find much support on this forum for these ideas. For those that have interest, the book Irreducible Mind: Toward a Psychology for the 21st Century is a well-written treatise on the topic.

A gentler step in that direction is to see what Michael Levin and his lab are up to. He is looking for (one aspect of) intelligence, and finding it at the cellular level and below, even in an agential version of bubble sort. He's certainly challenging the notion that consciousness is limited to brain cells. All of his findings arise through experimental observation, so it forces some reckoning in a way that sociological research doesn't.

> Perhaps it is not possible to simulate higher-level intelligence using a stochastic model for predicting text.

I think you're on to it. Performance is clustering because a plateau is emerging. Hyper-dimensional search engines are running out of steam, and now we're optimizing.

> Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better.

Aren't we the summation of intelligence from quintillions of beings over hundreds of millions of years?

Have LLMs really had more data?

By that argument, so are LLMs. They also wouldn't exist without all our ancestors.

No, by that argument, so would a can of soda.

To be smarter than human intelligence you need smarter than human training data. Humans already innately know right and wrong a lot of the time so that doesn't leave much room.

This is a very good point! I remember reading about AlphaGo and how they got better results training against itself vs training against historical human-played games.

So perhaps the solution is to train the AI against another AI somehow... but it is hard to imagine how this could extend to general-purpose tasks

> Humans already innately know

Gentle suggestion that there is absolutely no such thing as "innately know". That's a delusion, albeit a powerful one. Everything is driven by training data. What we perceive as "thinking" and "motivation" are emergent structures.

Innately as in you are born with it, the DNA learned not us humans. We have no clue how the DNA learned to think other than "survival of the fittest", and that is the oldest AI training method in the book.

> Human intelligence is markedly different from LLMs: it requires far fewer examples to train on, and generalizes way better.

That is because with LLMs there is no intelligence. It is Artificial Knowledge. AK not AI. So AI is AGI. Not that it matters for user-cases we have, but marketing needs 'AI' because that is what we were expecting for decades. So yeah, I also do not thing we will have AGI from LLMs - nor does it matter for what we are using it.