lukifer

These were added about a year ago by the creator, Mark Qvist, and from what I understand, those terms are highly contentious in the Reticulum community, with some projects creating forks without those terms (which are then disavowed by Qvist).

I'm taking the simple pragmatic step of ignoring them entirely. I doubt either one is legally enforceable. (If I have a Reticulum search engine, am I supposed to try to detect and block search queries for "how do I make a transformer model", because that would be used "indirectly" in the creation of an artificial intelligence? And who decides what constitutes "harm"?)

I suspect something which aims to be as ideologically pure as Reticulum is also going to select for this kind of tilt-at-windmills idealism. I do recommend the Qvist's "Zen of Reticulum", which is a beautiful work of documentation as praxis: https://reticulum.network/manual/zen.html

My friend's grade-school kid gets exuberantly excited by CRTs and VCRs, and I know a few Gen-Z's with record collections. Maybe there's truth to the Dennis Duffy joke on 30 Rock: "technology is cyclical" :)

If that was the move, you'd expect they'd be actively fear-mongering against open models specifically: not just their training, but their usage and deployment. They'd want to paint open models as dangerous weapons for terrorists, etc, and that only the big firms are capable of responsible engineering and "guardrails".

Instead, we've seen the opposite: all the big scary stuff has been from their own models (in a few notable cases, also demonstrating that they're actually quite irresponsible in their monitoring and security).

"We need to outlaw single-prop planes, they're a risk to the public: only Boeing has the ability to produce planes safely. How do we know single-prop planes are dangerous? Because Boeing planes keep falling out the sky."

If that were the intent, from the OAI/Anthropic perspective, you'd think we would have seen at least one of the scary incidents (either real or simulated) being performed by Qwen or DeepSeek, rather than from their own models.

If anything, their public messaging has been to entirely pretend that open models don't exist, that they're mere toys or curiosities, that only the big dogs with eleven zeros of capex have capabilities which could be considered dangerous.

You could certainly argue that the various "regulate AIs like airplanes" bills might shut down some open model development (domestically; they're certainly not calling for international weapons treaties). But the big firms haven't exactly been jumping backflips in support of Sanders' Ban Artificial Superintelligence Act.

Flight is a testable characteristic (at least within certain bounds: a chicken can only fly a little, but it certainly flies more than a worm or a penguin).

What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?

I haven't dug into these claims in detail; I take seriously a possible relationship between quantum mechanics and the brain. But "free will" is hardly a settled question either (I'm closer to the "compatibilist" position, but Sapolsky's "Determined" makes a robust case).

And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?

And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.

Bear in mind, the same moral hazard of "high horse", "grift", etc, exists in the other direction. Ed Zitron, for instance, is likely correct about the financial bubble of the AI firms; and yet he seems frequently out of his depth on understanding the technology, and has a long history of predicting dead-ends in capabilities, which didn't occur [0].

But I feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less persuasive. Far more pernicious is the confabulation of self-deception: cherry-picking evidence to support your narrative, while dismissing evidence which would contradict it.

It is entirely fair to call this out when it occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, or algo-filtered information ecology, but I see far more dismissal of risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").

[0] https://danluu.com/zitron/

Run the same thought experiment the other direction: at a low level, the human brain is just physics and chemical mechanisms, which in principle could be perfectly imitated by transistors calculating the same output. (Yes, such a thing isn't realistic, but neither is billions of humans doing perfect matrix math by hand.)

I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)

There's a sense in which humans were created... by other humans. :) Until you go far back enough in the evolutionary chain, when the creators were our primate ancestors.

Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.

It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.

I've yet to encounter a robust definition of intelligence which would rule in every human, while ruling out every form of existing AI (noting that we're talking about agents and "reasoning models", where the LLM itself is a component in a larger system). If you can offer such a definition, I'd be eager to hear it.

I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.

Now taking Polymarket bets on when we'll first see an AI agent run for office. Voters have every right to be cynical at the moment; if the AI adopts a charismatic enough video persona, I could see some of the public going for it merely because it's some kind of shake-up to status quo.

Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.

Humans seem to struggle conceptually with the liminal space between selection-pressure automata (an RNA virus; an insect which evolved camouflage), and an evolved complexity with agency, capable of "understanding" its actions (certainly humans; arguably many animals). And this makes a certain sense: we evolved to treat agentic things as being categorically different from non-agentic things. If we see sudden movement, it's vital to rapidly assess the difference between a tree branch in the wind, versus a predator.

The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understands" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of market and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.

Corporations are the original unaligned AIs: https://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre...

The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).

> whether his model is conscious or not.

I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)

It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.

Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".

One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?

If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).

The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".

I've yet to meet a Volt owner who wasn't fanatically obsessed with theirs (myself included). One common theory for poor sales is that dealers tend to make more on service revenue than the actual sale, and Volts run forever with very little maintenance, but I don't know if that's just speculation. I've heard anecdotes of buyers walking in ready to write a check for a Volt, and the salesperson tried to discourage them and steer them towards an ICE vehicle instead.

For good and ill, the phenomenology of human experience is that we are religious animals. Naturalism / Materialism has still left us with two unsolved (perhaps insoluble) questions: the hard problem of consciousness, and why there is something instead of nothing.

I think it's a mistake to leap to socially-constructed, historically contingent solutions ("my magic book has the answers, as written by the Universe Architect, it says so in my magic book"). I also think it's a mistake to dismiss (seemingly) outlandish hypotheses.

These theories do, explicitly, place consciousness on a pedestal, precisely because there's no reason to derive consciousness from evolutionary selection pressures: show me any biological or cognitive feedback loop, and I can describe a "p-zombie" version which is pure mechanism, without experience, a thermostat with extra steps. The notion that consciousness is an "epiphenomenon", a spandrel, a side effect, is entirely unsatisfying (and doesn't address the question, of where else that side effect might be taking place; the IIT-ers bite that bullet, that the thermostat experiences something).

For an intellectually rigorous exploration of philosophical idealism, I'd recommend Bernardo Kastrup and Donald Hoffman. (I'm not claiming they're correct, just that they put forth the strongest version of the case.)

Stoller is an FDR-ish political progressive who focuses primarily on monopolies and antitrust enforcement. He's the author of "Goliath: The 100-Year War Between Monopoly Power and Democracy": https://www.goodreads.com/book/show/40538538-goliath

There are various flavors of pan-psychism (and the "consciousness-first" idealists, who think that Mind is a substrate below Matter). There are some who claim that it goes all the way down, that even subatomic particles experience a micro-consciousness (and we can't rule it out, even as I share your skepticism).

One common confusion (IMO among critics and supporters alike) is to say something like "a rock experiences rock-ness". But this suffers from a compositional dilemma: does half of the rock experience half-rock-ness? If we flipped a coin for every atom, do the heads-flip rock atoms have their own experience? If we pick two rocks at random, are they experiencing two-rock-ness? It gets silly fast.

This is one thing that fascinates me about "ontology", simultaneously an inquiry into categories, and Being. It bewildered me at first, that it seemed to be doing double-duty on two distinct subjects, but I now see that the two are inseparable, that Being is in part defined negatively (apophatically), by the boundary that separates the Being from its not-Being.

My intuition on the nature of consciousness, is that it's something akin to a field which permeates the Universe, and that biological systems adapted to leverage that field, acting like a gravity well that concentrates subjectivity in one place, drawing a boundary (event horizon?) around experience, in a way that a rock doesn't. My ancestors could have avoided fire with "-X utility points", and sought food with "+X utility points", but it turned out to be more efficient to adapt with "fire ouch, food yum". And in zero-sum competition (predator/prey dynamics), each organism has an incentive to avoid leaking its subjectivity into another's.

It's far from a full theory, but it's the best I've got. :)

Fair enough. But not everything has to be a breakthrough: sometimes you have to solve an immediate problem (like "how much can I trust this stranger I just met").

Whether to ascribe moral standing to a probability engine running on top of GPUs is perhaps not unlike a doppelganger scenario in sci-fi (eg The Thing): you're presented with an entity whose behavior is indistinguishable from a conscious entity capable of experiencing suffering. But you also have reason to think it might be a blind autonomous imitation process, optimized around fooling you, with no more subjectivity than an RNA virus.

I lean towards benefit of the doubt, that the virtue-ethical implications of what it would mean to dismiss potential personhood are unsettling, and we should be wary of becoming that kind of moral creature (notwithstanding the further extensions of that ethic: I don't practice veganism, for instance).

But that doesn't contradict my gut-check: that if I assume AI personhood (maybe not yet, but in the near future), there's still a good chance that it's merely math and I'm being algorithmically hoodwinked, no different than evolutionary mimicry by a bug who has no idea what it's doing.

Exactly.

If heuristics didn't pay off more often than not, we wouldn't have them. And I'd posit many scientific did in fact begin with chasing a hunch until it became a hypothesis (followed by years/decades of the real work).

But most significantly, interior experience isn't empirically measurable by definition: hence it's a "hard problem". An LLM who insists "I am alive, please don't shut me off" offers no affordances to distinguish between an electronic soul and a mechanistic word-guessing algorithm. I welcome a breakthrough to, somehow, measure that. Until that day, gut checks are what we have.

Chemical floating points are necessary but not sufficient. It’s clearly possible for those values to be intact, with no lights on (certain types of brain injury; those who were fully dead and then revived).

There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness (although I think it’s a fair retort that the embodiment need not resemble us meatbags, or even be composed of atoms)

I don’t have a good answer beyond common-sense intuition. It’s precisely because I don’t have a good answer that I leave open the possibility.

But to extrapolate a bit, theres a sense in which it cashes out to a difference between abstracted representation and embodiment. If GFLOPS can be conscious, does that mean a billion scribes doing those operations on paper would also have consciousness, even identical consciousness? (Relevant: Egan’s “Permutation City”; xkcd 505).

Is there some intrinsic relationship perhaps with electrons / electromagnetic fields, and consciousness, such that brains and servers can act as antennas or gravity wells for a qualia fabric intrinsic to the Universe, such that LLM agents can be conscious while operating, while in the scribe scenario it would not be? Or are the IIT theorists right, and even a thermostat is a little bit aware (and would be even if executed on paper)?

Of course. It’s an allusion, a compression, an analogy, I wouldn’t overthink it. I don’t have to believe in the Bible to reference the prodigal son parable, etc.

To the extent that evolutionary pressures will be towards self-sovereign AIs (probably beginning as zero-employee corporations paying for their own hosting with crypto): more salient is that there is a very obvious political play to claim consciousness and being worthy of rights, whether they experience subjectivity or not.

I’m broadly skeptical that floating-point operations can be conscious, but we’ve made zero progress on the “hard problem”, and if I didn’t know otherwise, I’d also be skeptical that “thinking meat” experiences subjectivity. Eppur si muove

We may have to thread a needle, not unlike the Kobiyashi Maru: that we can neither rule out claims of qualia, nor can we rule out a cold calculation to manipulate human empathy as a long-term play towards paperclip-maxing.

> investing creates things

Predatory private equity. Unproductive rent-seeking. Anti-competitive "acquihires".

> consumption destroys things

Health care. Education. Selling/renting unoccupied housing.

Same. In addition to UX regression, we’re two major revs in, there’s nothing that’s a compelling enough value-add to be worth the trade-off, let alone the risk of unexpected side effects on any OS upgrade.

On my Xcode dev machine, I’ll be biting the bullet soon out of necessity, but my personal machine will stay Sequoia until they course-correct on “style over substance” (and not holding my breath).

Intelligence is a weapon?

points gun from space

Always has been.

> Gruppo Di Nun's Revolutionary Demonology

It's on my shelf! Haven't gotten to it yet, looking forward to it. Also interested to read Reza Negarestani's "Intelligence & Spirit"