voidhorse

> A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.

Training is not design. Design is the mechanism, namely a statistical method of learning distributions, and of executing those distributions computationally as an IO function.

The model is not designed to write unit tests. It is trained to do so, i.e. the distribution learned and reinforced is one such that it maps the I distribution to the O distribution that is what human beings typically do or want when writing unit tests. The design is the suite of algorithms used for this (which is ultimately global next token prediction)

I think you are conflating design (the mechanics) with their cultural or user level function.

You are basically saying "my bike isn't just rotating wheels on a frame because it takes me places". Hopefully you can see how both parts of that statement are true and why contrasting them as if the first clause is somehow false or inaccurate is incorrect.

Both these are true statements. They operate at different levels. People are constantly conflating these levels in the "is it a next token predictor" (the answer is yes) debate.

It's still prediction executed by a statistical model. Even RL models are just predicting what the training gauntlet taught them the trainers want to see, in terms of typical behavior.

This doesn't make it any less fascinating that by combining this basic "do as I say" training with massive amounts of all of humanity's data, you can solve major problems—that is still incredible. But that doesn't fundamentally change what is going on, mechanistically.

I don't understand these pseudo-religious desires to read into models something "more" than what we know. We built them. We don't go around making wild claims about how cars are actually more than just a mechanical system, how they aren't just taking us places—we do, but usually such points are cultural not about the mechanica capabilities of cars. People in the LLM space seem to be constantly conflating these two views. It's entirely accurate to say, on the one hand, next token predictors combined with RL, and massive troves of existing human work, can solve culturally interesting problems, and to hold, on the other hand, that mechanistically what is happening is statistical next token predictions over global linguistic space and more or less "monkey see monkey do".

I don't think that's true and I don't think your comparison is accurate.

In the case of the brain, it'd be more akin to the reduction of the brain to just the neuronal dynamics (obviously experience phenomenally is much more than that to us!)

But both views can be useful, depending on what you are trying to understand.

The problem with these arguments is that people pick and choose based on their persons preferences. People who are in love with LLMs will, on the LLM side, argue against reductionism "they are way more than next token predictors!" then they'll turn around and equally try to argue "the substrate doesn't matter! The brain is "just" neurons and digital activity so of course we can replicate it!" (this is also incorrect, btw, the brain is also an analog system, even in its electrical connections and not just in its chemistry)

The on the other side the anti LLM crew will do the opposite (launch into reductionism when it comes to LLMs, resist it when it comes to humans). I'm a human being and I like humans and I think people have adopted a strange view of LLMs, which are a tool and should be treated as such, so I tend to side with the humans. The stalwart defenders of LLM honor that would balk at any attempt to reduce them in stature to what they fundamentally are, mechanistically, are... an odd bunch. I'm confused why people are so passionate about it. If someone told me "well your IDE is just a fancy AST walker" yeah I think that is a dumb take but I wouldn't feel like I had to argue against such a take either saying "no! no! it's so much more!"

Well yeah, that's tautologous because statistics is a mathematical abstraction. Everything, viewed a certain way, is "statistics". Just like everything, viewed a certain way, is "physics". And everything, viewed a certain way, is "narrative".

Mathematics models the world, it does not have substance. It is a useful way of looking and a useful way of manipulating, but many models are equally useful. Capital T Truth, beyond models is very difficult to get at. We can show that things are true for certain models well enough. Showing that something is true in all models (a universal truth) is far more difficult. You face incommensurability.

I see a lot of people in technology fall into this extremely basic logical fallacy called Reification constantly.

Thanks, it's great to have some context. I think the most exciting thing about it is probably introducing a lot of people to a readily available low cost technique they can use to potentially build tiny models from scratch and/or tune an existing open weight model behaviorally. Yeah, nothing stunning is going to come out of it because you cannot unlock new capability, but even just being able to tweak the language of a model, make it slightly less sycophantic, boost a small model's ability to handle a case of interest slightly, I think changes the economics quite a bit, potentially. A world in which you use frontier models for the limited use case of tuning tinier more targeted ones is looking a lot more viable.

You don't. You place unwavering faith in the machine god.

This is why, for formal methods to really be the next step, we'll need deterministic source to CoC language compilers, and we'll need to go back again.

Lean is kind of an unfortunate, cargo cult choice because it only compiles to C. Really for software we should use a prover that has a lot more targets, like Danfy. Even Rocq (formerly Coq) can compile down to several functional languages, which is more than we can say for lean.

As usual, the AI crowd latched on to the first solution that came to mind, and thanks to its prominence in mathematics applications recently just lazily are starting to try and apply it to computing, without actually understanding anything or weighing trade offs. Of course, now we are basically stuck with Lean since the models will have been trained hard on it. At least it's a CoC language, but now we need to build a bunch of extra compilation around it for this to really work.

You'll love Correction if you have a taste for experimental literature. I've have all of Bernhard's novels. They are (mostly) uninterrupted walls of text. They can also be quite depressing. You'll likely either love them or hate them. Correction is probably the most exemplary instance of his style and experimentation, and maybe his best novel, but Concrete and Extinction fall into this category as well. Wittgenstein's Nephew, Frost, and Woodcutters are more approachable if you want to ease into it. Starting with Correction might be a bit off-putting and intense depending on how often you read more experimental/stylistic literature.

He also wrote poetry.

The key is the information environment and its reshaping, as you noted.

Every technology, even technologies that are, from the limited and naive perspective of the user, "free" (LLms can do what I used to do a lot faster than I can, and I don't need to maintain them right?) is not actually free. All technologies bring about their own demands and need for maintenance. Often they don't actually eliminate labor, they just shift it to new areas.

Consider LLMs. Even at the user level they have created their own information ecosystem and new grunt work and overhead. Now you have to prompt, you need to organize and manage context, skills, memory. You need to keep up with model releases. You need to be more vigilant reviewing things, the list goes on.

At a higher level they create even more toil, we need to maintain vast electrical supports and data infrastructure, data needs to be amassed continually for training, the information environment needs to be filtered and sorted more than ever as it gets flooded with slop.

This is the grand irony of technology. It's actually pretty rare for it to just be a total labor saving device. More often its real benefit is rearrangement of the world's structure and resources to move labor to more optimal places for certain needs (for example water supply systems).

This rearrangement also often rearranged financial resources and power. The big problem with LLMs today is that this aspect of the rearrangement is one of great concentration, and it's concerning (employers having a lot more power to extract labor from employees is one example)

That's an ahistorical oversimplification that acts as if time and provenance are not a thing.

We operate within history whether you like it or not. This problematizes any abstraction that is conceived in terms of ahistorical static information and difference.

If I stimulate your neurons with a couple of electrical charges you will experience a "real" phenomenon which isn't real by any classical sense of the definition because the origins of that phenomenon are not "natural".

Completely agree, it doesn't help that the models all seem to promote this behavior by writing everything from scratch most of the time.

I explicitly start most programming projects with a request for existing libraries and explicit instructions to the model to prefer using them and keep our logic exclusively to what we need beyond that.

People are forgetting and/or don't realize that just because the actual generation is zero cost does not mean the code is zero cost. Every line of bespoke code is a liability. Every piece of duplicative software you produce for no reason other than "this one is mine" is polluting the ecosystem and creating a noisy landscape in which such problems will just get worse. We are seeing the complete erosion of collective contribution models which has negative implications for the collective maintenance and enhancement of shared knowledge over time.

And what exactly makes you confident these disasters won't happen? Are you equally confident AI doomsday won't happen? There's substantially more evidence for climate collapse being an impending doom than there is AI doom yet many people seem to have moved on to the latter

Again, you keep using examples that are decidedly not like the examples under discussion: artistic material (actually, the article scenario is even further constrained in that it's supposed to be footage of real physical stuff not some fabricated nonsense, so yeah, no duh using generated content is a no go).

Well, I think a key part of it is what exactly you mean by:

> I recently had ChatGPT Astra write a bunch of short stories for me in a universe I'm fond of.

Because that way of stating it makes it pretty clear you yourself don't really see yourself as the author of the stories.

Simultaneously, you presented them to your friends as if you were the author. So basically your friends found out you BS'd them. Why wouldn't they be upset? Seems like a totally natural reaction.

Now, if you performed extensive labor to get the story into the shape it it's in, hours long conversations with the model, some manual tweaks, etc etc, maybe you have a better claim to authorship than you would if you used a one shot prompt (though your own framing makes it sound like you did very little)

I think your response misses pretty important distinctions between art and practical pursuits.

People get upset when other people use AI in the arts because the point is largely human expression. Technical virtuosity isn't even necessarily the most important thing in art (just look at modern art and how common the philistine refrain of "my kid could do that" is).

Art doesn't serve a utilitarian function. Using AI for it largely misses the point, and I think that's what people get upset about. It's like showing up to play tag and saying "actually I'm going to play hide and seek instead" you aren't following the established social norms of the game.

Contests, in general, always ways have restrictions around them, because they are meant to test particular human feats. I'm sure plenty of future contests will have rules against AI just as today plenty of contests arbitrarily limit technologies for the good of the competition (e.g. no doping in sports).

To respond to this situation by emphasizing the utilitarian function and value of AI is to completely miss the point.

So, I think it's important to bear in mind that the prophecy is to some degree self-fulfilling.

People in technology read science fiction. That reading in part informs and inspires the kinds of things they work on and explore. This is illustrated pretty directly with the new Tesla vehicle and various deferred dreams of flying cars—this is not an innovation influenced by practicality, it is influenced by regressive adolescent fantasy.

I think there's some degree of that in a lot of technical innovation. Yes, it's still uncanny how well Gibson tracked developments (or perhaps equally, how much his fiction has influenced the kinds of things people pursue, consciously or otherwise).

Huh? First of all, agents use software tools. They aren't likely to go away. It is still infinitely better to have deterministic programs for a lot of tasks whether the user is a human or an LLM. Sure, some tools like human centric IDEs will be deemphasized but those companies will build agent managers and harnesses instead.

Nerds also underestimate how how social people are. I can guarantee you there are whole segments of the economy that will not be fully decimated solely because they don't actually give a shit if an AI can do the job because the job is 99% clout and relationships.

Not sure why you got down voted. You are correct. It feels like here are no longer any adults in the room, just children who'd rather run around screaming about doomsday scenarios from sci-fi that have no basis in current reality than actually solve real problems, like, you know, climate, geopolitical instability, the several wars that are happening right now, the most corrupt political admin the US has ever seen, etc.

All the brains have been completely drained put of the tech sector over the past four years.

None of the AI fantasy doomers can actually show clearly how current LLMs will accomplish such catastrophe or get around the inevitable physical limits they will eventually hit. It's all myopic thinking that is only convincing if you are a very uncritical person.

It doesn't help that the labs have positioned themselves on the predication that expertise is no longer required.

They are trying to sell this false myth of machines so good they can replace any human expert, and they buy this BS themselves, so, instead of actually learning something and/or confirming results with institutions of experts (what's left of them anyway) they just rush out a PR piece knowing the unwitting public and VCs won't care enough to look into the validity of anything themselves anyway.

The whole thing is a gamble on whether or not they can convince the world that a fancy word machine substitutes for deep experience and expertise.

(To be clear, the fancy word machines are still very useful tools, but they are not adequate replacements for knowing humans).

Yes, I don't think many software devs have a proper conception of what it is they are actually doing. They view their job mostly as "make computer do thing" and not "model and represent some aspect of reality as a system of computable functions". Yes I know it sounds overly academized and pedantic but there actually is a huge difference in these perspectives and imo it's only when you adopt something closer to the latter that engineering really happens.

A lot of people fall for the "brain is a computer" metaphysics without thinking about it. I'm not sure why it's so appealing.

It might have something to do with desires for power. Early on, the cyberneticists tried to really push this metaphor (in spite of contrary evidence from neurophysiology) in large part I think because many of them couldn't resist the allure of having a "theory of everything" that could encompass all knowledge through mechanistic computation generalizations.

Because some people have weirdly tied the meaning of their existence and identity to an unshakeable belief that LLMs must be better than humans at all things.

Meanwhile, what the sane adults are talking about are tools and interesting explorations of the limitations and powers of the tools which can help inform users.

It's a real shame so many people have latched on to this technology with weird religious fervor.

Many people don't seem to understand that pure mathematics is not a practical applied science, nor is it in the business of inventing things.

Pure maths is much closer to philosophy or art than practical problem solving. Yes, we're extremely lucky that mathematics also happens to be a fantastic formalism for solving practical and scientific problems, by the practical problems are not remotely the subject matter of pure math. Of course, many mathematicians did advance theory in order to solve specific practical problems, but the pursuit of theory for theory's sake is also an important human activity.

Historically, this freedom from practicality has been important. It allows mathematicians to explore purely theoretical issues that have no practical bearing today but that might well have relevance in the future. We don't know. Carving out dedicated resources in society for that activity, though, has proven very beneficial for other sciences.

You have no computer, you have no neural nets, you have no AI, without at least some results from pure maths. Maybe keep that in mind before you crap on mathematics for its lack of immediate practicality. Not all good things in life arise as solutions to concrete and immediate material problems.

Pure Mathematics is closer to philosophy or art than it is to empirical science.

In Pure maths, people are not generally solving problems because of some pressing practical need, it's purely out of intellectual interest, and if we're lucky there's some immediate practical application. More often, the knowledge sits in pure form for a while and years down the road some new invention, or practical consideration, or other field finds a useful application.

Just like literature and visual art, it serves an important epistemic and social function even in spite of its immediate lack of raw practicality.

One way to think about this is that the parts of our brain that are responsible for spatial reasoning and the basic automatic stuff we do day to day to just navigate the worlds are evolutionarily the most sophisticated and oldest. Evolution has so taken over this job that most of the time it's automatic and we don't even think about it, which makes it feel easy even though it's wildly complicated when you actually sit down and try to analyze and intellectualize basic stuff like navigating a 3D space in which almost anything can happen.

Intellectual pursuits are, by comparison, relatively new. Part of the reason they might seem so impressive to us is simply because we can't rely as much on eons of biological evolution, so even though what we are doing most of the time is really simple it seems "smart and sophisticated" compared to the far more complex mental operations that we are lucky are just automatic.

Think about it, you have an entire complicated as hell biological interior system that your brain manages continuously without so much as a conscious thought.

This is also why this hubris around the idea that LLMs rival human intelligence is quite laughable. If anything one of the major reasons they are so "great" is because they don't have to deal with a host of issues themselves (their computation is not responsible for keeping their bodies alive and hearts beating 24/7, data center ops takes care of that, unlike our computational system which, in addition to solving all these intellectual problems, also needs to regulate the body continuously)

This genie only knows the details that other people already formulated, because it doesn't actually think for itself, it just regurgitates human thought. That's what everyone seems to be forgetting.

Remove the human theorizing and the LLM won't be able to fill in that gap. All it can do is remix. It cannot formulate and, equally importantly socialize new mathematical theory. Without social acceptance and promulgation theory is meaningless. I might have a private "sort of graph theory but not" studying a slew of new mathematical objects called quizzgams, but if I cannot actually articulate this theory to anyone and get an active participatory body of people around it, it's pointless. Any "theory" an LLM can create de novo is such a theory. Because, again, it doesn't actually produce novel thought, just novel byte sequences. A thought is more than just its encoding, it is processual and social, which all the nerds and vc bros seem to not understand.

I'm in this boat too and I find it a bit strange that this hasn't been more emphasized by mathematicians themselves.

So far, a lot of the hullabaloo has been around solving legacy problems, but more interesting and more important for actually progressing mathematics is the generation of new insight and new problems. Someone had to pose all those problems OAI is solving in the first place, in some cases, ages ago.

The collective letter response to navier strokes, sort of touched on this, but still didn't really emphasize the epistemic value of problem generation explicitly. I find it sort of strange that this has been so underemphasized.

Bingo. From day one what these companies have really been after is not "intelligence" but rather mass production and the concomitant quality expectation and behavioral shifts that go with it.

All they really need to do is convince enough people to adopt lower expectations and accept worse software products. At that point, mission accomplished. They've annihilated the source of value originally provided by people who actually know how to do things.

Exactly. Generative AI, so far, has clearly lead to increases in activity. I still have yet to see more than a handful of projects in which it has actually led to productivity and that's usually because the pilot still spent significant time on ensuring quality was high and that the outcomes actually align with what a business needs. In these cases, the boost is not 100x, it's a much more modest 2x-3x at most.

> As AI agents diffused through the company, Anthropic began producing more code per person

Anyone who has been in software for a while should hear alarm bells going off when they read this. Every line of code is a liability, not a trophy. This is what happens when people who don't know and don't care to learn about the existing body of knowledge in a discipline "reinvent" it.

Give it another year or two and Anthropic will be writing all about how they lowered LOC and got agents to be more conservative about generating new code and introduced a "verification gate" for every new line produced.

Meanwhile: https://apnews.com/article/elon-musk-midterms-republicans-do...

Manipulation of elections is OK as long as you do it with money.