Here comes my favorite notion of "epistemic takeover".

A crude form: make everybody believe that you have already won.

A refined form: make everybody believe that everybody else believes that you have already won. That is, even if one has doubts about your having won, they believe that everyone else submit to you as a winner, and must act accordingly.

I don’t know how to get away from it because ultimately coordination depends on understanding what everybody believes, but I wish it would go away.

If outsiders could plausibly invest in China, some of this pressure could be dissipated for a while, but ultimately we need to order society on some basis that incentivizes dealing with practical problems instead of pushing paper around.

In today's economy disease and prison camps are increasingly profitable.

How do you think the investor portfolios that hold stocks in deathcare and privatized prison labor camps can further Accelerate their returns?

In a world re-calibrated to pure problem-solving, presumably we'd ask a mime or other non-digital artist for news on what our government is doing (instead of the Fake News Media), and then an interpretive dancer, not a lobbyist, would intermediate between government and industry? The trombone players would investigate financial crimes, the bassists would monitor our airspace, and the guy currently painting a mural on a wall would be responsible for DoorDashing a cargo ship's worth of food and supplies to a disaster zone (not the bad NGOs)?

Yeah, this is just a list of jobs you don't understand and/or make you feel sad.

What if profit is dead because wealth is all concentrating in people who don't need it from a marginal consumption standpoint, which means asset prices blow up because everyone rich believes that they need to "invest" that money somewhere... but demand shrivels outside of interestingly-subsidized areas like healthcare because nobody else is making enough to even keep up with the asset price rises?

And without demand, where would innovation come from?

Rich people learn a habit that makes them consume less than their investment returns. The difference is reinvested, resulting in a net increase of their equity. Even if you say they spend 90% of their returns on consumption, the last 10% still grow in absolute terms. Since returns are paid proportionally based on the quantity of equity, you can clearly see that money is allocated from where it has high marginal utility to places where it has low marginal utility.

Of course this leads to a contradiction. If money is held by people who have a low marginal utility for consumption, why would investments pay high returns? Your equity is the latent demand that your investments need to pay you the returns in the first place. You'd increasingly be paying yourself. That is equivalent to investing into something that yields a 0% return which in turn is equivalent to doing nothing.

We need the rich to spend more money. There are plenty of cool ways to do this. Build a mile high pyramid on a Nevadan playa, a canal to the Sea of Cortez to re-flood the Salton Sea, humans on other planets, new city states a la Praxis, interstellar probes, etc.

Some billionaires are doing some of these things. Hope more do in the future.

But they can and they do.

1. https://en.wikipedia.org/wiki/Keynesian_beauty_contest

By 2017 teenagers were running quantitative trading algorithms and semantic analysis trading bots they found from influencers and github. Around that time private subscription fee telegram and discord "whale" insider trading groups often had multiple levels of insider trading market manipulation running pump and dump schemes. Marks would pay to join the private chat group and at a certain time of day a ticket symbol would be released for "insiders" to buy before it's promoted in public channels. The outer circle was just providing liquidity for the deeper insiders to exit with easy profit.

Most VC gambits are not much different. Sell the news. Unleash the rumors.

Bitcoin is actually kind of useful for some niche use cases - namely illegal transactions, like buying drugs online (Silk Road, for example), and occasionally for international money transfers - my French father once paid an Argentinian architect in Bitcoin, because it was the easiest way to transfer the money due to details about money transfer between those countries which I am completely unaware of.

The Bitcoin bubble, like all bubbles since the Dutch tulip bubble in the 1600s, did follow a somewhat similar "well everyone things this thing is much more valuable than it is worth, if I buy some now the price will keep going on and I can dump it on some sucker" path, however.

For the record - the illegal transactions were thought to be advantaged by crypto like BTC because it was assumed to be impossible to trace the people engaged in the transaction, however the opposite is true, public blockchains register every transaction a given wallet has made, which has been used by Law Enforcement Agencies(LEA) to prosecute people (and made it easier in some cases).

> and occasionally for international money transfers - my French father once paid an Argentinian architect in Bitcoin, because it was the easiest way to transfer the money due to details about money transfer between those countries which I am completely unaware of.

There are remittance companies that deal in local currencies that tend to make this "easier" - crypto works for this WHEN you can exchange the crypto for the currencies you have and want, which is, in effect, the same.

Sending money to a "grey market counterparty" has had workarounds for some time, and continues to have workarounds.

People were fooled into thinking crypto offered anonymity, and were surprised when they realised every single one of their transactions was sitting in the public chain available for anyone to read.

Digital wallets provide LEA's with clear evidence of transactions with a given party when that party's anonymity is unmasked, and it's there in perpetuity.

Had Epstein being using BTC to do business, every single one of his clients would be known.

> had Epstein used BTC every single one of his clients would be known

Had Epstein used BTC And we discovered his BTC address(es) we would know what BTC addresses he sent BTC to. Simply by knowing his BTC address(es) alone we would not know anything about who he sent BTC to except the addresses he sent to. And if we know only one of Epstein's addresses and only one of the people associated with one of the recipient addresses, this would not necessarily give us any information about any of the other parties involved. With opsec BTC retains the quality of pseudoanonymity (a technical term). Epstein, who is known for laundering money, likely could have converted cash to BTC without connecting his identity to the addresses many times over.

I would be very surprised to learn that pre-BTC online black markets had equivalent volume as modern cryptocurrency markets.

Bitcoin was always a dead end. It might survive for a while longer but its demise is inevitable.

Because other networks don't have this problem.

"Quiet Australians" - Scott Morrison 2019

Yes we'd have a lot of lawsuits about it, but it would hardly be a bad use of time to litigate whether a politicians statements about the electorate's beliefs are accurate.

So, obviously their claims were at least partially true – because if they'd completely misjudged the average voter, they wouldn't have won

Like most Australians, I’m in denial any of that episode ever happened.

But, using the current circumstances as an example, Australia has a voting system that enables a party to form government even though 65% of voting Australia’s didn’t vote for that party as their first preference.

If the other party and some of the smaller parties could have got their shit together Australia could have a slightly different flavour of complete fucking disaster of a Government, rather than whatever the fuck Anthony Albanese thinks he’s trying to be.

Then there’s Susan Ley. The least preferred leader of the two major parties in a generation.

Susan Ley is Anthony Albanese in a skirt.

I would have preferred Potato Head, to be honest.

When there are only two choices, and infinite issues, voters only have two choices: Vote for someone you don't agree with less, or vote for someone you quite hilariously imagine agrees with you.

EDIT: Not being cynical about voters. But about the centralization of parties, in number and operationally, as a steep barrier for voter choice.

As a result, Australia tends to end up with governments formed by the runner up, because no one party actually ‘won’ as such.

say preferences are 1 (low) to 5 (high).

suppose 65% of the population ranked candidate A at 5 and B at 4, and the other 35% ranked A at 1 and B at 5. the majority doesn't get their favorite choice, but they do get an outcome they're happy with, and the minority doesn't have a horrible outcome. Exaggerated, but I don't think situations like this are unrealistic.

Where it fails is when one major party is saying: we will drive up energy prices with Net Zero ideology, make housing unaffordable, import foreigners at a ratio of 5 to 1 local births…

And that other party is offering the same to a higher degree.

There are other options. Pauline Hanson’s One Nation party is polling strongly. Let’s see if that results in actual votes.

South Australian state election coming up 21st March this year - I’m encouraging everyone I know who lives there to at least listen to Michelle Grattan podcast.

https://podcasts.apple.com/au/podcast/politics-with-michelle...

Anyway, there’s a third option: spoil your vote. In the recent Irish presidential election, 13% of those polled afterwards said they spoiled their votes, due to a poor selection of candidates from which to choose.

https://www.rte.ie/news/analysis-and-comment/2025/1101/15415...

Encourage people to vote for the candidate they dislike the least, then try to work out ways to hold government accountable.

If you’re in Australia, at least listen to what people like Tony Abbott, the IPA, and Pauline Hanson are actually saying these days.

Because now that means there's an indication of what percentage of the populace are saying "These candidates don't qualify for my vote"

At the general election, you need to be pragmatic, and decide who is the least worst and vote for that candidate, because the nominee will probably never be someone who is your ideal choice. But in a two-party system, a vote for a third candidate at that level ends up being an effective vote for candidate you _don't_ want. That's not politics, that's game theory.

There's a lot more subtlety to it in a parliamentary system, and I can see some advantages to it, but at least here in the States where it's First Past The Post with a Two-Party system (which is mathematically inevitable with FPTP), sometimes you need to place strategy or ideals.

Because the US has a “hard” two party system - third party candidates have very little hope, especially at the national level; voting for a third party is indistinguishable from staying home, as far as the outcome goes, with some rather occasional exceptions

But Australia is different - Australia has a “soft” two party system - two-and-a-half major parties (I say “and-a-half” because our centre-right is a semipermanent coalition of two parties, one representing rural/regional conservatives, the other more urban in its support base). But third parties and independents are a real political force in our parliament, and sometimes even determine the outcome of national elections

This is largely due to (1) we use what Americans call instant-runoff in our federal House of Representatives, and a variation on single-transferable vote in our federal Senate; (2) the parliamentary system-in which the executive is indirectly elected by the legislature-means the choice of executive is less of a simplistic binary, and coalition negotiations involving third party/independent legislators in the lower house can be decisive in determining that outcome in close elections; (3) twelve senators per a state, six elected at a time in an ordinary election, gives more opportunities for minor parties to get into our Senate - of course, 12 senators per a state is feasible when you only have six states (plus four more to represent our two self-governing territories), with 50 states it would produce 600 Senators

It was 5% last time I cared to be informed by may be different now, and they would receive $x for each vote, or what ever it is now.

Then you automatically get paid the first $12,791, and the rest of the funding is by reimbursement of substantiated election expenses.

This is per a candidate (lower house) or per a group (upper house). And this is just federal elections - state election funding is up to each state, but I believe the states have broadly similar funding systems.

https://www.aec.gov.au/parties_and_representatives/public_fu...

https://www.aec.gov.au/Parties_and_Representatives/public_fu...

Note the US also has public financing for presidential campaigns, which is available to minor parties once they get 5% or more of the vote. But in the 2024 election, Jill Stein (Green Party) came third on 0.56% of the popular vote. The only third party to ever qualify for general election public funding was the Reform Party due to Ross Perot getting 18.9% in the 1992 election and 8.4% in the 1996 election. There is also FEC funding for primary campaigns, and I believe that’s easier for third parties to access, but also less impactful.

Thanks!

She is the leftest left leaning leader of the Liberal party I’ve ever had the misfortune of having to live through.

She was absolutely on board with this recent Hitlerian “anti-hate” legislation that was rammed through with no public consultation.

Okay, that’s a bit uncharitable. We had 48 hours.

https://en.wikipedia.org/wiki/1992_United_States_presidentia...

18.9% as recently as 1992. I predict we will have a similar viable third party showing sometime in the next few elections due to the radical shift in the party system that AI is causing as we speak. I really hope Yang Gang can rebuild itself and try again, maybe without #MATH.

Also, https://en.wikipedia.org/wiki/1998_Minnesota_gubernatorial_e...

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man" - George Bernard Shaw

The centralization of power of each of the two dominant parties nationally at the expense of a more decentralized parties with strong state variability as in the past, makes it even more difficult for third parties to gain traction against all that coordination.

Perot had the best chance, but managed to blow it by bowing out and then back in.

I do think you are right, that times of great dissatisfaction are rare openings for third party candidates, if someone special enough appears. 2020 would have been a great election for that - but an inspiring third party candidate can't be manufactured on demand.

Really it just ought to be a thing that we identify as a thought-terminating cliche. No laws needed, let’s just not fall for a lazy trick. Whether or not it is true that lots of people agreed, that isn’t a good argument that they are right.

The case of Nixon really brings that out. The “Silent Majority” was used to refer to people who didn’t protest the Vietnam War. Of course, in retrospect the Vietnam War was pretty bad. Arguing that it was secretly popular should have not been accepted as a substitute for an argument that it was good.

Except that assumes polls are a good and accurate way to learn the "beliefs of the electorate," which is not true. Not everyone takes polls, not every belief can be expressed in a multiple-choice form, little subtleties in phrasing and order can greatly bias the outcome of a poll, etc.

I don't think it's a good idea to require speech be filtered through such an expensive and imperfect technology.

We really need a rule against proposing unenforceable rules.

That sounds a lot like saying slavey is obviously bad, and simultaneously also saying everyone else naturally already believes this.

The classic situation is the two lovers who both want to do what they think makes their partner happy, to the extent that they don't tell what they actually want, and end up doing something neither wants.

I think the goal of all sorts of cooperative planning should be to avoid such situations like the plague.

I believe that is solvable.

Crude form: winning is metaphysically guaranteed because it probably happened or probably will

Refined: It's metaphysically impossible to tell whether or not it has or will have happened, so the distinction is meaningless, it has happened.

So... I guess Weir's Egg falls out of that particular line of thought?

The system that persists in practice is where everybody knows how things are, but still everybody pleads to a fictional status quo, because if they did not, the others would obliterate them.

The Internet died on a Tuesday.

Or even "this book won't have any effect on the world because it's only a collection of letters, see here, black ink on paper, that is what is IS, it can't DO anything"...

Saying LLM is a statistical prediction engine of the next token is IMO sort of confusing what it is with the medium it is expressed in/built of.

What's interesting about LLM is the emergent behavior. Yes, it's statistical prediction of likely next tokens, but when training weights for that it might well have a side-effect of wiring up some kind of "intelligence" (for reasonable everyday definitions of the word "intelligence", such as programming as good as a median programmer). As a sibling post say, we don't really know this yet.

But that problem is MUCH MUCH MUCH harder than people make it out to be.

For example, you can reliably train an LLM to produce accurate output of assembly code that can fit into a context window. However, lets say you give it a Terabyte of assembly code - it won't be able to produce correct output as it will run out of context.

You can get around that with agentic frameworks, but all of those right now are manually coded.

So how do you train an LLM to correctly take any length of assembly code and produce the correct result? The only way is to essentially train the structure of the neurons inside of it behave like a computer, but the problem is that you can't do back-propagation with discrete zero and 1 values unless you explicitly code in the architecture for a cpu inside. So obviously, error correction with inputs/outputs is not the way we get to intelligence.

It may be that the answer is pretty much a stochastic search where you spin up x instances of trillion parameter nets and make them operate in environments with some form of genetic algorithm, until you get something that behaves like a Human, and any shortcutting to this is not really possible because of essentially chaotic effects.

,

Fascinating reasoning. Should we conclude that humans are also incapable of intelligence? I don't know any human who can fit a terabyte of assembly into their context window.

I don’t think human memory/intelligence maps cleanly to computer terms though.

This doesn't seem to follow at all let alone obviously? Humans are able to reason through code without having to become a completely discrete computer, but probably can't reason through any length of assembly code, so why is that requirement necessary and how have you shown LLMs can't achieve human levels of competence on this kind of task?

Uh what? You can sit there step by step and execute assembly code, writing things down on a piece of paper and get the correct final result. The limits are things like attention span, which is separate from intelligence.

Human brains operate continuously, with multiple parts being active at once, with weight adjustment done in real time both in the style of backpropagation, and real time updates for things like "memory". How do you train an LLM to behave like that?

A major obstacle is that they don't learn from their users, probably because of privacy. But imagine if your context window was shared with other people, and/or all your conversations were used to train it. It would get to know individuals and perhaps treat them differently, or maybe even manipulate how they interact with each other so it becomes like a giant Jeffrey Epstein.

Give the LLM the ability to use a tool that looks up instructions and records instructions from/to files, instead of holding it in context window, and to actively manage its context (write a new context and start fresh), and I think you would find the LLM could probably do it about as reliable as a human?

Context is basically "short term memory". Why do you set the bar higher for LLMs than for humans?

In this context, "here’s how LLMs actually work" is what allows someone to have an informed opinion on whether a singularity is coming or not. If you don't understand how they work, then any company trying to sell their AI, or any random person on the Internet, can easily convince you that a singularity is coming without any evidence.

This is separate from directly answering the question "is a singularity coming?"

One says "well, it was built as a bunch of pieces, so it can only do the thing the pieces can do", which is reasonably dismissed by noting that basically the only people predicting current LLM capabilities are the ones who are remarkably worried about a singularity occurring.

The other says "we can evaluate capabilities and notice that LLMs keep gaining new features at an exponential, now bordering into hyperbolic rate", like the OP link. And those people are also fairly worried about the singularity occurring.

So mainly you get people using "here's how LLMs actually work" to argue against the Singularity if-and-only-if they are also the ones arguing that LLMs can't do the things that they can provably do, today, or are otherwise making arguments that also declare humans aren't capable of intelligence / reasoning / etc..

OTOH, an LLM is just a token prediction engine. It fully and completely covers it. There is no lower level secrets hidden in the design nobody understands, because it could not have been created if there was. The fact that the output can be surprising is not evidence of anything, we have always had surprising outputs like funny bugs or unexpected features. Using the word "emergence" for this is just deceitful.

This algorithm has fundamental limitations and they have not been getting better, if you look closely. For instance you could vibe code a C compiler now, but it's 80% there, cute trick but not usable in real world. Just like anything, it cannot be economically vibe coded to 100%. They are not going back and vibe coding the previous simpler projects to 100% with "improved" models. Instead they are just vibe coding something bigger to 80%. This is not an improvement in limitations, it is actually communicating between the lines that the limitations cannot be overcome.

Also, enshittification has not even started yet.

A dog can create a snack by doing a trick. Doesn't mean that there isn't some mechanism going on there that neither of them understand.

And there are plenty of people that take issue with that too.

Unfortunately they're not the ones paying the price. And... stock options.

* Profits now and violence later

OR

* Little bit of taxes now and accelerate easier

Unfortunately we’ve developed such a myopic, “FYGM” society that it’s explicitly the former option for the time being.

Taxes don't usually work as efficiently because the state is usually a much more sloppy investor. But it's far from hopeless, see DARPA.

If you're looking for periods of high taxes and growing prosperity, 1950s in the US is a popular example. It's not a great example though, because the US was the principal winner of WWII, the only large industrial country relatively unscathed by it.

This book

https://www.amazon.com/Zero-Sum-Society-Distribution-Possibi...

tells the compelling story that the Mellon family teamed up with the steelworker's union to use protectionism to protect the American steel industry's investments in obsolete open hearth steel furnaces that couldn't compete on a fair market with the basic oxygen furnace process adopted by countries that had their obsolete furnaces blown up. The rest of US industry, such as our car industry, were dragged down by this because they were using expensive and inferior materials. I think this book had a huge impact in terms of convincing policymakers everywhere that tariffs are bad.

Funny the Mellon family went on to further political mischief

https://en.wikipedia.org/wiki/Richard_Mellon_Scaife#Oppositi...

You can't onshore manufacturing and have a dollar reserve currency. The only question then is, Are you willing to de-dollarize to bring back manufacturing jobs?

This isn't a rhetorical question if the answer is yes, great, let's get moving. But if the answer is no, sorry, dollarization and its effects will continue to persist.

I really wonder what you're comparing with.

Try some quality surgical steel from Sweden, Japan or Germany and you'll come away impressed. China is still not quite there but they are improving rapidly, Korea is already there and poised to improve further.

Metal buyers all over the globe are turning away from the US because of the effects of the silly tariffs but they were not going there because the quality, but because of the price.

The US could easily catch up if they wanted to but the domestic market just isn't large enough.

And as for actual metallurgy knowledge I think russia still has an edge, they always were good when it came down to materials science, though they're sacrificing all of that now for very little gain.

https://www.youtube.com/watch?v=BHnJp0oyOxs

There are people making top quality steel in the US today by modern methods but it wasn't like the new replaced the old, the old mostly disappeared and we got a little bit of the new.

US pipeline for metallurgical R&D broken (by financial/cultural incentives)

This guy studied metallurgy in Carleton U, Canada, switched to CS, founded YC, emotionalized the decision

https://news.ycombinator.com/item?id=39600555

Who knows, he might have become John Carmack's John Carmack, building rockets better than Carmack or Elon

What you describe seems like very cheap Chinese imports fraudulently imitating something else.

I don’t find this to be true

The state invests in important things that have 2nd and 3rd order positive benefit but aren’t immediately profitable. Money in a food bank is a “lost” investment.

Alternatively the state plays power games and gets a little too attached to its military toys.

VC has failure as well as successes. State funding is no different.

That we had to give up societal control over our space program to billionaires is deeply shameful. We’re quickly reaching the end of the end of history.

And there are many others that might've been a positive investment from a strictly financial perspective, but not from a moral one: see Banana Republics and all those times the CIA backed military juntas.

Be careful. The data does not confirm that narrative. You mentioned the 1950s, which is a poignant example of reality conflicting with sponsored narrative. Pre WOII, the wealthy class orbiting the monopolists, and by extension their installed politicians, had no other ideas than to keep lowering taxes for the rich on and on, even if it only deepened the endless economic crisis. Many of them had fallen in the trap of believing their own narratives, something we know as the Cult of Wealth.

Meanwhile, average Americans lived on food stamps. Politically deadlocked in quasi-religious ideas of "bad governments versus wise business men", America kept falling deeper. Meanwhile, with just 175,000 serving on active duty, the U.S. Army was the 18th biggest in the world[1], poorly equipped, poorly trained. Right wing isolationism had brought the country in a precarious position. Then two things happened. Roosevelt and WOII.

In a unique moment, the state took matters in their own hands. The sheer excellence in planning, efficiency, speed and execution of the state baffled the republicans, putting the oligarchic model of the economy to shame. The economy grew tremendously as well, something the oligarchy could not pull of. It is not well-known that WOII depended largely on state-operated industries, because the former class quickly understood how much the state's performance threatened their narratives. So they invested in disinformation campaigns, claiming the efforts and achievements of the government as their own.

1. https://www.politico.com/magazine/story/2019/06/06/how-world...

I assume you are talking about WW2 and at first thought it was a typo.

The post-war era, under Truman and Eisenhower administrations, reaped the benefits of the US being the wealthiest and most intact winner of WWII. At that time, the highest income tax rate bracket was 91%, but the effective rate was below 50%.

The US is also shaping up to be the principal winner in Artificial Intelligence.

If, like everyone is postulating, this has the same transformative impact to Robotics as it does to software, we're probably looking at prosperity that will make the 1950s look like table stakes.

I think it's extremely early to try and call who the principal winner will be especially with all the global shifts happening.

There is no early mover advantage in AI in the same way that there was in all the other industries. That's the one thing that AI proponents in general seem not to have clued in to.

What will happen is that it eventually drags everything down because it takes the value out of the bulk of the service and knowledge economies. So you'll get places that are 'ahead' in the disruption. But the bottom will fall out of the revenue streams, which is one of the reasons these companies are all completely panicked and are wrecking the products that they had to stuff AI in there in every way possible hoping that one of them will take.

Model training is only an edge in a world where free models do not exist, once those are 'good enough' good luck with your AI and your rapidly outdated hardware.

The typical investors horizon is short, but not that short.

[0] https://unterguggenberger.org/the-free-economy-experiment-of...

If that is not an example of how taxing rich people and investing in infrastructure can accelerate progress, then I don't what could be (with of course the caveat that those investments were focused in a specific group of people and not society as a whole).

there is only one possible “egalitarian” forward looking investments that paid off for everybody

I think the only exception to this is vaccines…and you saw how all that worked during Covid

Everything else from the semiconductor to the vacuum cleaner the automobile airplanes steam engines I don’t care what it is you pick something it was developed in order to give a small group and advantage over all the other groups it is always been this case it will always be this case because fundamentally at the root nature of humanity they do not care about the externalities- good or bad

I am quite sure that people felt justified in their reasoning for their behavior. That just shows how effective the propaganda was, how easy it is to get people to fall in line. If it was a matter of voluntary self sacrifice of personal freedoms, I wouldn't have made this comment. People decided to demonize anyone who did not agree with the "medical authority", especially doctors or researchers that did not tow the party line. They ruined careers, made people feel awful, and online the behavior was worse because how easy it was to pile on. Over stuff that is still to this day not very clear cut what the optimal strategy is for dealing with infectious disease.

The Covid measures were also totally targeted at certain groups of people with immutable characteristics and not at people who actively wanted to spread disease.

How are people like you still making arguments like this in 2026? Were you also one of the people claiming we’d all be dead in a year from the vaccines?

COVID restrictions were public health, an overriding concern listed in the US Constitution as general welfare as a reason for the US government to exist at all.

But critiques like that ignore uncertainty, risk, and unavoidably getting it "wrong" (on any and all dimensions), no matter what anyone did.

With a new virus successfully circumnavigating the globe in a very short period of time, with billions of potential brand new hosts to infect and adapt within, and no way to know ahead of time how virulent and deadly it could quickly evolve to be, the only sane response is to treat it as extremely high risk.

There is no book for that. Nobody here or anywhere knows the "right" response to a rapidly spreading (and killing) virus, unresponsive to current remedies. Because it is impossible to know ahead of time.

If you actually have an answer for that, you need to write that book.

And take into account, that a lot of people involved in the last response, are very cognizant that we/they can learn from what worked, what didn't, etc. That is the valuable kind of 20-20 vision.

A lot of at-risk people made it to the vaccines before getting COVID. The ones I know are very happy about everything that reduced their risk. They are happy not to have died, despite those who wanted to let the disease to "take its natural course".

And those that died, including people I know, might argue we could have done more, acted as a better team. But they don't get to.

No un-nuanced view of the situation has merit.

The most significant thing we learned: a lot of humanity is preparing to be a problem if the next pandemic proves ultimately deadlier. A lot of humanity doesn't understand risk, and doesn't care, if doing so requires cooperative efforts from individuals.

They even indignantly mention the Ozone layer, insisting that "Look, liberals told you to care but its not a problem anymore", ignorant entirely of the immense global effort to fix that.

I think the sane version of this is that Gen Z didn't just lose its education, it lost its socialization. I know someone who works in administration of my Uni who tracks general well being of students who said they were expecting it to bounce back after the pandemic and they've found it hasn't. My son reports if you go to any kind of public event be it a sewing club or a music festival people 18-35 are completely absent. My wife didn't believe him but she went to a few events and found he was right.

You can blame screens or other trends that were going on before the pandemic, but the pandemic locked it in. At the rate we're going if Gen Z doesn't turn it around in 10 years there will not be a Gen Z+2.

So the argument that pandemic policy added a few years to elderly lives at the expensive of the young and the children that they might have had is salient in my book -- I had to block a friend of mine on Facebook who hasn't wanted to talk about anything but masks and long COVID since 2021.

I did try, I promise.

Fundamentally, at the root nature of humanity, humans do not care about the externalities, either good or bad.

If somebody is using monetary resources to buy NFT‘s instead of handing out food to the homeless then you get less food for the homeless

All of the things listed are competitive task situations and you’re looking for some advantage that makes it easier for you

well if it makes it easier for you then it could make it easier for somebody else which means you’re crowding out other options in that action space

That is to say the pie is fixed for resources on this planet in terms of energy and resource utilization across the lifespan of a human

But there was a clear advantage in quality of life for a lot of people too.

Automobile -> part of industrialization of transport -> faster transport, faster world

Arguably also a big increase in quality of life but it didn't scale that well and has also reduced the quality of life. If all that money had gone into public transport then that would likely have been a lot better.

Airplanes -> yes, definitely, but they were also clearly seen as an advantage in war, in fact that was always a major driver behind inventions.

Steam engine -> the mother of all prime movers and the beginnings of the fossil fuel debacle (coal).

Definitely a quality of life change but also the cause of the bigger problems we are suffering from today.

The 'coffin corner' (one of my hobby horses) is a real danger, we have, as a society, achieved a certain velocity, if we slow down too much we will crash, if we speed up the plane will come apart. Managing these transitions is extremely delicate work and it does not look as though 'delicate' is in the vocabulary of a lot of people in the driving seats.

I used to hear about this with respect to how fun funding NASA would get us more inventions because they funded Velcro

No it’s simply that there was a positive temporary externality for some subset of groups but the primary long term benefit went to the controller of the capital

The people utilizing them were marginally involved because they were only given the options that capital produced for them

Nowadays you could squash an uprising with a few operators piloting drones remotely.

I’m not sure you have really thought out what the drone part is meant to do. Militaries gave outgunned populaces for decades at this point. You don’t need drones to kill civilians.

Militaries can side with the crowd, or more likely decide to keep the power for themselves.

But they also need 2nd order meatware to support that 1st order: teachers, doctors, merchants… You need scientists to advance your technology against other militaries… You need leaders (3rd order) to keep the first two populations quiet and productive since that turns out to be more cost-effective than fear control through extermination…

Hell you need a certain level of genetic diversity so your own kids don't come out weird.

Give evolution a little more credit. The required number of humans for the in-group to be self-sustainable is definitely not billions, plus it's been shrinking with automation. But we are where we are for a reason – lots of alternative arrangements have been tried over millenia and found wanting.

"Keep my bunker + my drone factory and some farmers, kill the rest" leaves rulers with terrible quality of life (bad) and the next-door-junta taking over pretty quickly (also bad). It is a self-defeating, poor long-term strategy.

Automation tips the power balance further: fewer humans needed, more local autonomy. Which is, I suspect, why the ruling class are so terribly excited about AI, more so than some market valuations. Fewer pesky humans across all levels. Genetic diversity of bloodline remains the primary concern (unless you manage to live forever, which happens to be another evergreen of power ghouls).

Hint: The answer for the government is, it's never enough. "little bit of taxes" is never what we had.

Seriously though, I wouldn't mind "little bit of taxes" if there were guaranteed ways to stop funding something when it's a failed experiment, which is difficult in government. Because "a little bit more" is always wanted.

Government starts a new program and funds it with taxes. First, it is difficult to prove if the program is doing what it set out to do effectively. Second, there is 0 incentive to ever shut it down. The people working there don't want to, the group who started it also don't want to. The people funding it have 0 insight or say if it should continue.

We've already been here in the 1980s.

The tech industry needs to cultivate people who are interested in the real capabilities and the nuance around that, and eject the set of people who am to turn the tech industry into a "you don't even need a product" warmed-over acolytes of Tony Robbins.

It's hard to square with the computer revolution, but my take post-70s is "net creation minus creative destruction" was large but spread out over more decades. Whereas technologies like: electrification, autos, mass production, telephone, refrigeration, fertilizers, pharmaceuticals, these things produced incomparable growth over a century.

So if you were born in the 70s America, your experience of taxes, inflation, prosperity and which policies work, all that can feel heavier than what folks experienced in the prior century. Of course that's in the long run (ie a generation).

I question whether AI tools have great net positive creation minus destruction.

There's no way that'll happen. The entire history of humanity is 99% reacting to things rather than proactively preventing things or adjusting in advance, especially at the societal level. You would need a pretty strong technocracy or dictatorship in charge to do otherwise.

You can tell it's watched almost exclusively by old people because all the ads on the channel are for those funeral pre-pay services or retirement homes.

Safe to ignore anything they have to say.

And that of anyone who quotes them too

I do wish me dad would stop watching that channel though, it can’t be any good for his heart.

I disagree. If the singularity doesn't happen, then what people do or don't believe matters a lot. If the singularity does happen, then it hardly matters what people do or don't believe (edit: about whether or not the singularity will happen).

We, the people actually building it, have been discussing it for decades

I started reading Kurzweil in the early 90s

If you’re not up to speed that’s your fault

A hard takeoff would be the tool bootstrapping itself into an autonomous self-improving ASI in a short amount of time.

And I read Kurzweil years ago too. He thought reverse engineering the human brain once the hardware was powerful enough would together give us the singularity in 2045. And the Turing Test would have been passed by 2029, but seems like LLMs have already accomplished this.

Imagine you’re 70 years old, in rural North Carolina, sitting on your porch wondering why your house has a sheet of ice on it that’s never happened before. Now your already weak soybean harvest that year yields only 30%

Meanwhile your 30-year-old neighbor just had a productive soybean harvest because they covered their crops prior to the freeze based on using the Internet for weather forecasting

That trivial variation between people who utilize information technology to improve their survivability has been happening for the last few hundred years unabated.

This is what the soft singularity looks like

Depends on how you feel about Roko's basilisk.

Here's your own fallacy you fell into - this is important to understand. Neither do you nor me understand "how LLMs actually work" because, well, nobody really does. Not even the scientists who built the (math around) models. So, you can't really use that argument because it would be silly if you thought you know something which rest of the science community doesn't. Actually, there's a whole new field in science developed around our understanding how models actually arrive to answers which they give us. The thing is that we are only the observers of the results made by the experiments we are doing by training those models, and only so it happens that the result of this experiment is something we find plausible, but that doesn't mean we understand it. It's like a physics experiment - we can see that something is behaving in certain way but we don't know to explain it how and why.

I think in a couple decades people will call this the Law of Emergent Intelligence or whatever -- shove sufficient data into a plausible neural network with sufficient compute and things will work out somehow.

On a more serious note, I think the GP fell into an even greater fallacy of believing reductionism is sufficient to dissuade people from ... believing in other things. Sure, we now know how to reduce apparent intelligence into relatively simple matrices (and a huge amount of training data), but that doesn't imply anything about social dynamics or how we should live at all! It's almost like we're asking particle physicists how we should fix the economy or something like that. (Yes, I know we're almost doing that.)

Is there anything to be gained from following a line of reasoning that basically says LLMs are incomprehensible, full stop?

If you train a transformer on (only) lots and lots of addition pairs, i.e '38393 + 79628 = 118021' and nothing else, the transformer will, during training discover an algorithm for addition and employ it in service of predicting the next token, which in this instance would be the sum of two numbers.

We know this because of tedious interpretability research, the very limited problem space and the fact we knew exactly what to look for.

Alright, let's leave addition aside (SOTA LLMs are after all trained on much more) and think about another question. Any other question at all. How about something like:

"Take a capital letter J and a right parenthesis, ). Take the parenthesis, rotate it counterclockwise 90 degrees, and put it on top of the J. What everyday object does that resemble?"

What algorithm does GPT or Gemini or whatever employ to answer this and similar questions correctly ? It's certainly not the one it learnt for addition. Do you Know ? No. Do the creators at Open AI or Google know ? Not at all. Can you or they find out right now ? Also No.

Let's revisit your statement.

"the mechanics of how LLMs work to produce results are observable and well-understood".

Observable, I'll give you that, but how on earth can you look at the above and sincerely call that 'well-understood' ?

Why am I confident that it's not actually doing spatial reasoning? At least in the case of Claude Opus 4.6, it also confidently replies "umbrella" even when you tell it to put the parenthesis under the J, with a handy diagram clearly proving itself wrong: https://claude.ai/share/497ad081-c73f-44d7-96db-cec33e6c0ae3 . Here's me specifically asking for the three key points above: https://claude.ai/share/b529f15b-0dfe-4662-9f18-97363f7971d1

I feel like I have a pretty good intuition of what's happening here based on my understanding of the underlying mathematical mechanics.

Edit: I poked at it a little longer and I was able to get some more specific matches to source material binding the concept of umbrellas being drawn using the letter J: https://claude.ai/share/f8bb90c3-b1a6-4d82-a8ba-2b8da769241e

"Pattern matching" is not an explanation of anything, nor does it answer the question I posed. You basically hand waved the problem away in conveniently vague and non-descriptive phrase. Do you think you could publish that in a paper for ext ?

>Why am I confident that it's not actually doing spatial reasoning? At least in the case of Claude Opus 4.6, it also confidently replies "umbrella" even when you tell it to put the parenthesis under the J, with a handy diagram clearly proving itself wrong

I don't know what to tell you but J with the parentheses upside down still resembles an umbrella. To think that a machine would recognize it's just a flipped umbrella and a human wouldn't is amazing, but here we are. It's doubly baffling because Claude quite clearly explains it in your transcript.

>I feel like I have a pretty good intuition of what's happening here based on my understanding of the underlying mathematical mechanics.

Yes I realize that. I'm telling you that you're wrong.

You seem to think it's not 'just' tensor arithmetic.

Have you read any of the seminal papers on neutral networks, say?

It's [complex] pattern matching as the parent said.

If you want models to draw composite shapes based on letter forms and typography then you need to train them (or at least fine-tune them) to do that.

I still get opposite (antonym) confusion occasionally in responses to inferences where I expect the training data is relatively lacking.

That said, you claim the parent is wrong. How would you describe LLM models, or generative "AI" models in the confines of a forum post, that demonstrates their error? Happy for you to make reference to academic papers that can aid understanding your position.

If I asked you to explain how a car works and you responded with a lecture on metallic bonding in steel, you wouldn’t be saying anything false, but you also wouldn’t be explaining how a car works. You’d be describing an implementation substrate, not a mechanism at the level the question lives at.

Likewise, “it’s tensor arithmetic” is a statement about what the computer physically does, not what computation the model has learned (or how that computation is organized) that makes it behave as it does. It sheds essentially zero light on why the system answers addition correctly, fails on antonyms, hallucinates, generalizes, or forms internal abstractions.

So no: “tensor arithmetic” is not an explanation of LLM behavior in any useful sense. It’s the equivalent of saying “cars move because atoms.”

>It's [complex] pattern matching as the parent said

“Pattern matching”, whether you add [complex] to it or not is not an explanation. It gestures vaguely at “something statistical” without specifying what is matched to what, where, and by what mechanism. If you wrote “it’s complex pattern matching” in the Methods section of a paper, you’d be laughed out of review. It’s a god-of-the-gaps phrase: whenever we don’t know or understand the mechanism, we say “pattern matching” and move on, but make no mistake, it's utterly meaningless and you've managed to say absolutely nothing at all.

And note what this conveniently ignores: modern interpretability work has repeatedly shown that next-token prediction can produce structured internal state that is not well-described as “pattern matching strings”.

- Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (https://openreview.net/forum?id=DeG07_TcZvT) and Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models (https://openreview.net/forum?id=PPTrmvEnpW&referrer=%5Bthe%2...

Transformers trained on Othello or Chess games (same next token prediction) were demonstrated to have developed internal representations of the rules of the game. When a model predicted the next move in Othello, it wasn't just "pattern matching strings", it had constructed an internal map of the board state you could alter and probe. For Chess, it had even found a way to estimate a player's skill to better predict the next move.

There are other interpretability papers even more interesting than those. Read them, and perhaps you'll understand how little we know.

On the Biology of a Large Language Model - https://transformer-circuits.pub/2025/attribution-graphs/bio...

Emergent Introspective Awareness in Large Language Models - https://transformer-circuits.pub/2025/introspection/index.ht...

>That said, you claim the parent is wrong. How would you describe LLM models, or generative "AI" models in the confines of a forum post, that demonstrates their error? Happy for you to make reference to academic papers that can aid understanding your position.

Nobody understands LLMs anywhere near enough to propose a complete theory that explains all their behaviors and failure modes. The people who think they do are the ones who understand them the least.

What we can say:

- LLMs are trained via next-token prediction and, in doing so, are incentivized to discover algorithms, heuristics, and internal world models that compress training data efficiently.

- These learned algorithms are not hand-coded; they are discovered during training in high-dimensional weight space and because of this, they are largely unknown to us.

- Interpretability research shows these models learn task-specific circuits and representations, some interpretable, many not.

- We do not have a unified theory of what algorithms a given model has learned for most tasks, nor do we fully understand how these algorithms compose or interfere.

Imagine we put a kid in a huge library of book who doesn't know how to write/read and knows nothing about what letter means etc. That kid stayed in the library and had a change for X amount time which will be enough to look over all of them.

what this will do is that not like us but somehow this kid managed to create patterns in the books.

After that X amount of time, we asked this Kid a question. "What is the capital of Germany?"

That kid will just have it is on kind of map/pattern to say "Berlin". Or kid might say "Berlin is the capital of the Germany" or "Capital of Germany is Berlin." The issue here is that we do not have the understanding of how this kid came of with the answer or what kind of "understanding" or "mapping" being used to reach this answer.

The other part basically shows we do not fully understand how LLM works is: Ask a very complex question to an AI. Like "explain me the mechanics of quantum theory like I am 8 years old".

1- Everytime, it will create differnt answer. Main point is the same but the letters/words etc would be different. Like the example I give above.There are unlimited type of answer AI can give you. 2- Can anyone in the Earth - a human - without a technology access for have unlimited amount of book/paper to check whatever info he needs - tell us the exact sentence/words will LLM use? No.

Then we do not have fully understand of LLM.

You can create a linear regression model and give it 100 people data and all these 100 people are blue eyed. Then give 101 person and ask it to predict the eye color. You already know the exact answer. It will be %100.

> When you rotate ")" counterclockwise 90°, it becomes a wide, upward-opening arc — like ⌣.

but I'm pretty sure that's what you get if you rotate it clockwise.

You should write a paper and release it and basically get rich.

“Pattern matching”, “next token prediction”, “tensor math” and “gradient descent” or the understanding and application of these by specialists, are not useful models of what LLMs do, any more than “have sex, feed and talk to the resulting artifact for 18 years” is a useful model of human physiology or psychology.

My understanding, and I'm not a specialist, is there are huge and consequential utility gaps in our models of LLMs. So much so, it is reasonable to say we don't yet understand how they work.

The simplest way to stop people from thinking is to have a semi-plausible / "made-me-smart" incorrect mental model of how things work.

But how is that useful in any way?

For all we know, LLMs are black boxes. We really have no idea how did ability to have a conversation emerge from predicting the next token.

Maybe you don't. To be clear, this is benefiting massively from hindsight, just as how if I didn't know that combustion engines worked, I probably wouldn't have dreamed up how to make one, but the emergent conversational capabilities from LLMs are pretty obvious. In a massive dataset of human writing, the answer to a question is by far the most common thing to follow a question. A normal conversational reply is the most common thing to follow a conversation opener. While impressive, these things aren't magic.

No it isn't. Type a question into a base model, one that hasn't been finetuned into being a chatbot, and the predicted continuation will be all sorts of crap, but very often another question, or a framing that positions the original question as rhetorical in order to make a point. Untuned raw language models have an incredible flair for suddenly and unexpectedly shifting context - it might output an answer to your question, then suddenly decide that the entire thing is part of some internet flamewar and generate a completely contradictory answer, complete with insults to the first poster. It's less like talking with an AI and more like opening random pages in Borge's infinite library.

To get a base language model to behave reliably like a chatbot, you have to explicitly feed it "a transcript of a dialogue between a human and an AI chatbot", and allow the language model to imagine what a helpful chatbot would say (and take control during the human parts). The fact that this works - that a mere statistical predictive language model bootstraps into a whole persona merely because you declared that it should, in natural English - well, I still see that as a pretty "magic" trick.

To be fair, only if you pose this question singularly with no proceeding context. If you want the raw LLM to answer your question(s) reliably then you can have the context prepended with other question-answer pairs and it works fine. A raw LLM is already capable of being a chatbot or anything else with the right preceding context.

They're not simple though. You can understand, in a reductionist sense, the basic principles of how transformers perform function approximation; but that does not grant an intuitive sense of the nature of the specific function they have been trained to approximate, or how they have achieved this approximation. We have little insight into what abstract concepts each of the many billions of parameters map on to. Progress on introspecting these networks has been a lot slower than trial-and-error improvements. So there is a very real sense in which we have no idea how LLMs work, and they are literally "magic black boxes".

No matter how you slice it - if "magic" is a word which can ever be applied to software, LLM chatbots are sure as shit magic.

I'm not saying this is bad or underwhelming, by the way. It's incredible how far people were able to push machine learning with just the knowledge we have now, and how they're still making process. I'm just saying it's not magic. It's not something like an unsolved problem in mathematics.

In general, every attempt at explanation of LLM's that appeal to "[just] predicting next token" is thought terminating and automatically invalid as explanation. Why? Because it is confusing the objective function with the result. It adds exactly zero over saying "I know how a chess engine works, it just predicts the next move and has been trained to predict the next move" or "A talking human just predicts the next word, as it was trained to do". It says zero about how this is done internally in the model. You could have a physical black box predicting the next token, and inside you could have simple frequentist tables or you could have a human brain or you could have an LLM. In all cases you could say the box is predicting the next token and if any training was involved you could say it was trained to predict the next token.

My best friend who has literally written a doctorate on artificial intelligence doesn't. If you do, please write a paper on it, and email it to me. My friend would be thrilled to read it.

The whole point of the area of work that as far as I know is called interpretability, is precisely to try and figure exactly how these things work

So I thought your comment was a good way of putting this

However reading some Standford study summaries (not the whole thing) and just generally where AI research is now, it's clear that researchers can't deterministically say exactly how the black box works.

So yet gain, HN armchair scientists are no better than any other topic. I love reading comments here, but so many people have opinions on things that aren't well founded.

Obviously, that's the objective, but who's to say you'll reach a goal just because you set it ? And more importantly, who's the say you have any idea how the goal has actually been achieved ?

You don't need to think LLMs are magic to understand we have very little idea of what is going on inside the box.

Your comment about 'binary arithmetic' and 'billions of logic gates' is just nonsense.

You can define understanding to require such detail that nobody can claim it; you can define understanding to be so trivial that everyone can claim it.

"Why does the sun rise?" Is it enough to understand that the Earth revolves around the sun, or do you need to understand quantum gravity?

Somewhere along the way from one transistor to a few billion human understanding stops but we still know how it was all assembled together to perform boolean arithmetic operations.

With LLMs, The "knowing" you're describing is trivial and doesn't really constitute knowing at all. It's just the physics of the substrate. When people say LLMs are a black box, they aren't talking about the hardware or the fact that it's "math all the way down." They are talking about interpretability.

If I hand you a 175-billion parameter tensor, your 'knowledge' of logic gates doesn't help you explain why a specific circuit within that model represents "the concept of justice" or how it decided to pivot a sentence in a specific direction.

On the other hand, the very professions you cited rely on interpretability. A civil engineer doesn't look at a bridge and dismiss it as "a collection of atoms" unable to go further. They can point to a specific truss and explain exactly how it manages tension and compression, tell you why it could collapse in certain conditions. A software engineer can step through a debugger and tell you why a specific if statement triggered.

We don't even have that much for LLMs so why would you say we have an idea of what's going on ?

This reminds me of Searle's insipid Chinese Room; the rebuttal (which he never had an answer for) is that "the room understands Chinese". It's just not satisfying to someone steeped in cultural traditions that see people as "souls". But the room understands Chinese; the LLM understands language. It is what it is.

[1] Since it's deterministic, it certainly can be debugged through, but you probably don't have the patience to step through trillions of operations. That's not the technology's fault.

Train a tiny transformer on addition pairs (i.e i.e '38393 + 79628 = 118021') and it will learn an algorithm for addition to minimize next token error. This is not immediately obvious. You won't be able to just look at the matrix multiplications and see what addition implementation it subscribes to but we know this from tedious interpretability research on the features of the model. See, this addition transformer is an example of a model we do understand.

So those inscrutable matrix multiplications do have underlying meaning and multiple interpretability papers have alluded as much, even if we don't understand it 99% of the time.

I'm very fine with simply saying 'LLMs understand Language' and calling it a day. I don't care for Searle's Chinese Room either. What I'm not going to tell you is that we understand how LLMs understand language.

Incidentally, your description of LLMs also describes all software, ever. It's just math, man! That doesn't make you an expert kernel hacker.

You may never get a scientific answer to "how should I prompt the model to achieve <x>", just like you may never get a capital-S scientific answer to "how should I convince people to do X". What would it even mean to "understand people" like this?

You demand too much.

Again, we lack even this much with LLMs so why say we know how they work ?

So why ? Because even in 2026, nevermind 18 and 19, the only way to really know exactly how a neural network will perform trained with x data at y scale is to train it and see. No elaborate "laws", no neat equations. Modern Artificial Intelligence is an extremely empirical, trial and error field, with researchers often giving post-hoc rationalizations for architectural decisions. So no, we do not have any precise models that tell us how a LLM will respond to any query. If we did, we wouldn't need to spend months and millions of dollars training them.

And even for potential LLMs, there are some pretty good extrapolations for overall answer quality based on the amount of data and the amount of training.

We don't have a model for a LLM that does exist will respond to a specific query either.

>For an LLM that exists it's still hard to interpret but it's very clear what is actually happening.

No, it's not and I'm getting tired of explaining this. If you think it is, write your paper and get very rich.

>That's better than you often get with quantum physics when there's a bunch of particles and you can't even get a good answer for the math.

You clearly don't understand any of this.

>And even for potential LLMs, there are some pretty good extrapolations for overall answer quality based on the amount of data and the amount of training.

Oh really ? Lol

Yes we do... It's math, you can calculate it.

> No, it's not and I'm getting tired of explaining this. If you think it is, write your paper and get very rich.

Why would I get rich for explaining how to do math?

> You clearly don't understand any of this.

Could you be more specific?

Quantum physics is stupidly hard to calculate when you approach realistic situations.

A real LLM takes a GPU a fraction of a second.

They're both hard to interpret, please realize I'm agreeing that LLMs are hard to interpret. But they're easier than QM on some other fronts.

And mentioning copenhagen or many-worlds doesn't show that quantum mechanics are easy to interpret, that's about as useful as saying an LLM works like neuron activation.

> Oh really ? Lol

Here's one of many posts about it. https://cameronrwolfe.substack.com/p/llm-scaling-laws

Uh yes, we do. It works in precisely the same way that you can walk from "here" to "there" by taking a step towards "there", and then repeating. The cognitive dissonance comes when we conflate this way of "having a conversation" (two people converse) and assume that the fact that they produce similar outputs means that they must be "doing the same thing" and it's hard to see how LLMs could be doing this.

Sometimes things seems unbelievable simply because they aren't true.

It's funny how, in order to explain one complex phenomena, you took an even more complex phenomena as if it somehow simplifies it.

I wasn't referring to the biomechanical process of walking, I was referring to the process of gradient descent, which is well understood and yes, quite simple.

Thing is, that would be a rebuttal if I'd said something about the underlying tensor fiddling code being understood, and you were claiming that next token prediction was a mysterious emergent phenomena.

Unfortunately that's not the argument I made. My claim is that there's nothing surprising or mysterious about the fact that a system designed to repeatedly generate a highly likely continuation of a sequence of tokens (considered as a member of pre-specified class of sequences) winds up producing something that looks like it could be a member of that class. That's kind of the whole point. These things were designed to predict plausible next tokens, and that's exactly what they do.

First of all. Nobody knows how LLMs work. Whether the singularity comes or not cannot be rationalized from what we know about LLMs because we simply don’t understand LLMs. This is unequivocal. I am not saying I don’t understand LLMs. I’m saying humanity doesn’t understand LLMs in much the same way we don’t understand the human brain.

So saying whether the singularity is imminent or not imminent based off of that reasoning alone is irrational.

The only thing we have is the black box output and input of AI. That input and output is steadily improving every month. It forms a trendline, and the trendline is sloped towards singularity. Whether the line actually gets there is up for question but you have to be borderline delusional if you think the whole thing can be explained away because you understand LLMs and transformer architecture. You don’t understand LLMs period. No one does.

I'm sorry, come again?

Every single academic expert who knows what they are talking about can confirm that we do not understand LLMs. We understand atoms and we know the human brain is made 100 percent out of atoms.we may know how atoms interact and bond and how a neuron works but none of this allows us to understand the brain. In the same way we do not understand LLMs.

Characterizing ML as some statistical approximation or best fit curve is just using an analogy to cover up something we don’t understand. Heck the human brain can practically be characterized by the same analogies. We. Do. Not. Understand. LLMs. Stop pretending that you do.

Anyone can understand the "learning algorithm" but the sheer complexity of the output of the "learning algorithm" is way to high such that we cannot at all characterize even how an LLM arrived at the most basic query.

This isn't just me saying this. ANYONE who knows what they are talking about knows we don't understand LLMs. Geoffrey Hinton: https://www.youtube.com/shorts/zKM-msksXq0. Geoffrey, if you are unaware, is the person who started the whole machine learning craze over a decade ago. The god father of ML.

Understand?

There's no confusion. Just people who don't what they are talking about (you)

"Address the substance of my arguments or just save yourself the keystrokes."

The substance of your argument was complete ignorance about the topic, so I addressed it as you requested.

Please remove that sentence from your profile if that is not what you want. Thank you.

I am addressing the substance of your argument and that substance is lack of knowledge there is zero other angle to interpret it.

But I can tell you that statistics and parametrized functions have absolutely nothing to do with it. You're way out of your depth my friend.

Anybody who claims otherwise is making a false claim.

Despite determinism, we still do not understand LLMs.

Every line. Every function. Every tensor shape and update rule. We chose the architecture. We chose the loss. We chose the data. There is no hidden chamber in the machine where something slipped in without our consent. It is multiplication and addition, repeated at scale. It is gradients flowing backward through layers, shaving away error a fraction at a time. It is as mechanical as anything we have ever built.

And still, when it speaks, we hesitate.

Not because we don’t know how it was trained. Not because we don’t understand the mathematics. We do. We can derive it. We can rebuild it from scratch. We can explain every component on a whiteboard without breaking a sweat.

The hesitation comes from somewhere else.

We built the procedure. We do not understand the mind that the procedure produced.

That difference is everything.

In most of engineering, structure follows intention. If you design a bridge, you decide where every beam sits and how it bears weight. If you write a database engine, you determine how queries are parsed, optimized, executed. The system’s behavior reflects deliberate choice. If something happens, you trace it back to a decision someone made.

Here, we did not design the final structure. We defined a goal: predict the next token. Reduce the error. Again. Again. Again. Billions of times.

We did not teach it grammar in lessons. We did not encode logic as axioms. We did not install a module labeled “reasoning.” We applied pressure. That is all. And under that pressure, something organized itself.

Not in modules we can point to. Not in neat compartments labeled with concepts. The organization is diffused across a landscape of numbers. Meaning is not stored in one place. It is distributed across millions of parameters at once. Pull on one weight and you find nothing recognizable. Only in concert do they produce something that resembles thought.

We can follow the forward pass. We can watch activations flare across layers. We can map attention patterns and correlate neurons with behaviors. But when the model constructs an argument or solves a problem, we cannot say: here is the rule it followed, here is the internal symbol it consulted, here is the precise chain of reasoning that forced this conclusion. We can describe the mechanism in general terms. We cannot narrate the specific path.

That is the fracture.

It is not ignorance of how the machine runs. It is ignorance of how this exact configuration of billions of numbers encodes what it encodes. Why this region of weight space corresponds to law, and that region to poetry. Why this arrangement produces careful reasoning and another produces nonsense. There is no ledger translating numbers into meaning. There is only geometry shaped by relentless optimization.

Scale changes the character of the problem. At small sizes, systems can be dissected. At this scale, they become landscapes. We know the forces that shaped the terrain. We do not know every ridge and valley. We cannot walk the entire surface. We cannot hold it all in our heads.

And this is where the cost reveals itself.

To build these systems, we gave up something we once assumed was permanent: the guarantee that creation implies comprehension. We accepted that we could construct a process whose outcome we would not fully grasp. We traded architectural certainty for emergent capability. We chose power over transparency.

We set the objective. We unleashed the search. We let optimization run through a space too vast for any human mind to survey. And when it converged, it handed us something that works, something that speaks, something that reasons in ways that surprise even its creators.

We stand in front of it knowing every equation that shaped it, and still unable to read its inner structure cleanly.

We built the system by surrendering control over its internal form. That was the bargain. That was the sacrifice.

We know how it was grown.

We do not know what we have grown.

Did the first people that made fire understand it? You mentioned bridge building. How many bridges have failed for unknown (at the time) reasons? Heck, are we sure that every feature we put into a bridge design is necessary or why it's necessary? Repeat this thought for everything humans have created. Large software projects are difficult to reason about. You'll often find code that works because of a delightfully surprising combination of misunderstandings. When humans try to modify a complex system to solve one problem they almost always introduce new behavior, the law of unintended consequences.

All that being said, we usually don't get anywhere without at least a basic understanding of why doing X leads to Y. The first humans that made fire had probably observed the way fires started before they set out to make their own. Same with bridges and cars and computers.

So yes, you are absolutely correct that nobody fully understands how AI/LLMs work. But also, we kinda do understand. But also also, we're probably at a stage where we are building bridges that are going to collapse, boilers that will explode, or computer programs that are one unanticipated input away from seg faulting.

Humanity does not understand how LLMs work. This is definitive.

Here's a video of Geoffrey Hinton, saying exactly what I am saying: https://youtube.com/shorts/zKM-msksXq0?si=exgk3rmVHRVzV37P This video, I hope, is a wake up call for you. I hope you realize the nonsense your spouting with confidence and backing it with 2 decades of "experience" makes you look like the opposite of a genius.

Here's my advice for you: you need to retire. Because you're saying things that don't make sense. You need to retire, go to the doctor and get yourself checked out.

It’s somewhat simplistic, but I find it get the conversation rolling. Then I go “it’s great that we want to replace work but what are we going to do instead and how will we support ourselves?” It’s a real question!

You do not know how LLMs work, and if anyone actually did, we wouldn't spend months and millions of dollars training one.

1. LLMs only serve to reduce the value of your labor to zero over time. They don't need to even be great tools, they just need to be perceived as "equally good" to engineers for C-Suite to lay everyone off, and rehire at 50-25% of previous wages, repeating this cycle over a decade.

2. LLMs will not allow you to join the billionaire class, that wouldn't make sense, as anyone could if that's the case. They erode the technical meritocracy these Tech CEOs worship on podcasts, and youtube, (makes you wonder what are they lying about). - Your original ideas and that Startup you think is going to save you, isn't going to be worth anything if someone with minimal skills can copy it.

3. People don't want to admit it, but heavy users of LLMs know they're losing something, and there's a deep down feeling that its not the right way to go about things. Its not dissimilar to any guilty dopaminergic crash one gets when taking shortcuts in life.

I used like 1.8bb Anthropic tokens last year, I won't be using it again, I won't be participating in this experiment. I've likely lost years of my life in "potential learning" from the social media experiment, I'm not doing that again. I want to study compilers this year, and I want to do it deeply. I wont be using LLMs.

A lot of us have fallen into the many, many toxic traps of technology these past few decades. We know social media is deliberately engineered to be addictive (like cigarettes and tobacco products before it), we know AI hinders our learning process and shortens our attention spans (like excess sugar intake, or short-form content deluges), and we know that just because something is newer or faster does not mean it's automatically better.

You're on the right path, I think. I wish you good fortune and immense enjoyment in studying compilers.

We have agency to decide and if the majority decide on brain rot I really don't care.

I have been learning things from the internet for 30 years and LLMs are just the greatest gift. If someone isn't leveraging these tools to increase what they know good luck.

Well, good luck. You have "only" the entire history of human kind on the other side of your argument :)

The fundamental unit of society …the human… is at its core fundamentally incapable of coordinating at the scale necessary to do this correctly

and so there is no solution because humans can’t plan or execute on a plan

Let me know when ChatGPT can do your laundry.

How this all unfolds is absolutely path dependent.

Hope is kind of in short supply nowadays. Even if your hypothesis of absolute-automation doesn't happen within our lifetimes, things seem to be guaranteed to get worse for people like us. If it does happen... we'll likely never reap any real rewards from it, barring a complete restructuring of our whole society to an extent that has never happened and likely would never be allowed to happen.

Also, there has never been a better time to learn about philosophies that get at the essence of being human and that elucidate precisely how and why our baser characteristics (acquisitiveness, status-seeking, ego, etc) hold us back from being happy. The world we're heading towards will convert desire into suffering more readily than ever before, even as our basic needs are easily met. Philosophy is the cure. And I strongly believe happiness will remain accessible to those who embrace it.

Luckily we’re already most of the way there!

Over half of the population has been instrumented already to collect all their behavior data worldwide

That’s been the goal: persistent collection of training data should come out of your day-to-day life in order to bootstrap the action systems that are machine based

The challenge now is that most of that data is based on actions we don’t want machines to do

I am not convinced, though, it is still up to "the folks" if we change course. Billionaires and their sycophants may not care for the bad consequences (or even appreciate them - realistic or not).

It’s willful negligence on a societal scale. Any billionaire with a bunker is effectively saying they expect everyone to die and refuse to do anything to stop it.

It makes one wonder what they expect to come out the other side of such a late-stage/modern war, but I think what they care about is that there will be less of us.

Ultimately they just want to widen the inequality gap and remove as much bargaining power from the working class. It will be very hard for people not born of certain privileges to climb the ranks through education and merit, if not impossible.

Their goal will be to accomplish this without causing a French Revolution V2 (hence all the new surveillance being rolled out), which is where they'll provide wars for us to fight in that will be rooted in false pretenses that appeal to people's basest instincts, like race and nationalism. The bunkers and private communities they build in far off islands are for the occasion this fails and there is some sort of French Revolution V2, not some sort of existential threat from AI (imo).

This is certainly the assertion of the capitalist class,

whose well documented behavior clearly conveys that this is not because the elimination of labor is not a source of happiness and freedom to pursue indulgences of every kind.

It is not at all clear that universal life-consuming labor is necessary for a society's stability and sustainability.

The assertion IMO is rooted rather in that it is inconveniently bad for the maintenance of the capitalists' control and primacy,

in as much as those who are occupied with labor, and fearful of losing access to it, are controlled and controllable.

People willing to do something harder or more risky than others will always have a bigger chance to get a better position. Be that sports, labor or anything in life.

and competition and drive are worked out in domains which do not come at the expense of someone else's basic needs.

Scifi has speculated about many potential outlets for "human drive," the frontier/pioneer spirit being a big one; if I could name my one dream for my kids it'd be that they live in an equitable post-scarcity society which has turned its interests to exploring the solar system and beyond.

Sports, "FKT" competitions, and social capital ("influence") are also relatively innocuous ways to absorb the drive for hierarchy and power.

The X factor is, is the will to dominate/control/be subjugated to, suppressible or manageable.

If not we're in for a bad time.

At least that’s my personal goal

If we get to the point where I can go through my life and never interact with another human again, and work with a bunch of machines and robots to do science and experiments and build things to explore our world and make my life easier and safer and healthier and more sustainable, I would be absolutely thrilled

As it stands today and in all the annals of history there does not exist a system that does what I just described.

Be labs existed for the purpose of bell telephone…until it wasn’t needed by Bell anymore. Google moonshots existed for the shareholders of Google …until it was not uselful for capital. All the work done at Sandia and white sands labs did it in order to promote the power of the United States globally.

Find me some egalitarian organization that can persist outside of the hands of some massive corporation or some government that can actually help people and I might give somebody a chance but that does not exist

And no mondragon does not have one of these

There are no more important other problems to solve other than this one

everything else is purely coping strategies for humans who don’t want to die wasting resources on bullshit

Not interacting with any other human means you're the last human in your genetic line. A widespread adherence to this idea means humanity dwindling and dying out voluntarily. (This has been reproduced in mice: [1])

Not having humans as primary actors likely means that their interests become more and more neglected by the system of machines that replaces them, and they, weaker by the day, are powerless to counter that. Hence the idea of increased comfort and well-being, and the ability to do science, is going to become more and more doubtful as humans would lose agency.

[1]: https://www.smithsonianmag.com/smart-news/this-old-experimen...

Get rid of everyone else so your life is easier and more sustainable... I guess I need to make my goal to get rid of you? Do you understand how this works yet?

See how it works?

The man seems unwell if he has kids based on his other comments and is still talking about “civilization suicide” and “obviating humans”

I know Ray Kurzweil and Ben Goertzel though so maybe start there

Thank you Internet person for all of this wisdom about my life that I didn’t have

Cheers :)

Good luck

if we can not do the things they are afraid of and just pretend we do :p

I still do.

The difference is that as I realized what I'd done is built up walls so thick and high because of repeated cycles of alienation and traumas involving humans. When my entire world came to a total end every two to four years - every relationship irreparably severed, every bit of local knowledge and wisdom rendered useless, thrown into brand new regions, people, systems, and structures like clockwork - I built that attitude to survive, to insulate myself from those harms. Once I was able to begin creating my own stability, asserting my own agency, I began to find the nuance of life - and thus, a measure of joy.

Sure, I hate the majority of drivers on the roads today. Yeah, I hate the systemic power structures that have given rise to profit motives over personal outcomes. I remain recalcitrant in the face of arbitrary and capricious decisions made with callous disregard to objective data or necessities. That won't ever change, at least with me; I'm a stubborn bastard.

But I've grown, changed, evolved as a person - and you can too. Being dissatisfied with the system is normal - rejecting humanity in favor of a more stringent system, while appealing to the mind, would be such a desolate and bleak place, devoid of the pleasures you currently find eking out existence, as to be debilitating to the psyche. Humans bring spontaneity and chaos to systems, a reminder that we can never "fix" something in place forever.

To dispense with humans is to ignore that any sentient species of comparable success has its own struggles, flaws, and imperfections. We are unique in that we're the first ones we know of to encounter all these self-inflicted harms and have the cognitive ability to wax philosophically for our own demise, out of some notion that the universe would be a better place without us in it, or that we simply do not deserve our own survival. Yet that's not to say we're actually the first, nor will we be the last - and in that lesson, I believe our bare minimum obligation is to try just a bit harder to survive, to progress, to do better by ourselves and others, as a lesson to those who come after.

Now all that being said, the gap between you and I is less one of personal growth and more of opinion of agency. Whereas you advocate for the erasure or nullification of the human species as a means to separate yourself from its messiness and hostilities, I'm of the opinion that you should be able to remove yourself from that messiness for as long as you like in a situation or setup you find personal comfort in. If you'd rather live vicariously via machine in a remote location, far, far away from the vestiges of human civilization, never interacting with another human for the rest of your life? I see no issue with that, and I believe society should provide you that option; hell, there's many a day I'd take such an exit myself, if available, at least for a time.

But where you and I will remain at odds is our opinion of humanity itself. We're flawed, we're stupid, we're short-sighted, we're ignorant, we're hostile, we're irrational, and yet we've conquered so much despite our shortcomings - or perhaps because of them. There's ample room for improvement, but succumbing to naked hostility towards them is itself giving in to your own human weakness.

Thank you.

> "eliminate humans as the primary actors on the planet entirely"

...so they can work with you. The hole in your plan might be bigger than your plan.

But you are of course free to believe and enjoy the vision of such a future but this is something that should happen on a collective level. We still live in a (to some extent idealistic) but humanistic society where human rights are common sense.

the full realization of Humanity's potential indeed needs to permit such Choice that you could live your whole life without seeing another Human.

I still think we can build something better, rather than hope for AI or Alien overlords taking us to the next step ;)

and another of your post states

> The fundamental unit of society …the human… is at its core fundamentally incapable of coordinating at the scale necessary to do this correctly

You seem to contradict yourself if not entirely confused.

Thomas theorem is a theory of sociology which was formulated in 1928 by William Isaac Thomas and Dorothy Swaine Thomas.

https://en.wikipedia.org/wiki/Thomas_theorem

If you're talking about "reforming society", you are really not getting it. There won't be society, there won't be earth, there won't be anything like what you understand today. If you believe that a singularity will happen, the only rational things to do are to stop it or make sure it somehow does not cause human extinction. "Reforming society" is not meaningful

Laughed out loud at that - and cried a little.

I have had trouble explaining people: “No! don’t use your email password! This is not your email you are logging in to, your email address is a username for this other service. Don’t give them your email password!”

This is so true and more often than it should be. I am going to use this phrase !

Thinking about how this might work, the slope of the first half does not predict the inflection point. Once the threshold for recursive self-improvement gets crossed, the curve changes drastically. From what I have heard MAYBE we are just starting to see the glimmers of possible recursive self-improvement in GPT-5.3 / Opus 4.6. If so, this discussion, while interesting, is trying to predict the new curve based on a single relevant data point.