syllogism

I think LLMs are net helpful if used well, but there's also a big problem with them in workplaces that needs to be called out.

It's really easy to use LLMs to shift work onto other people. If all your coworkers use LLMs and you don't you're gonna get eaten alive. LLMs are unreasonably effective at generating large volumes of stuff that resembles diligent work on the surface.

The other thing is, tools change trade-offs. If you're in a team that's decided to lean into static analysis, and you don't use type checking in your editor, you're getting all the costs and less of the benefits. Or if you're in a team that's decided to go dynamic, writing good types for just your module is mostly a waste of time.

LLMs are like this too. If you're using a very different workflow from everyone else on your team, you're going to end up constantly arguing for different trade-offs, and ultimately you're going to cause a bunch of pointless friction. If you don't want to work the same way as the rest of the team just join a different team, it's really better for everyone.

Software to date has been a [Jevons good](https://en.wikipedia.org/wiki/Jevons_paradox). Demand for software has been constrained by the cost efficiency and risk of software projects. Productivity improvements in software engineering have resulted in higher demand for software, not less, because each improvement in productivity unblocks more of the backlog of projects that weren't cost effective before.

There's no law of nature that says this has to continue forever, but it's a trend that's been with us since the birth of the industry. You don't need to look at AI tools or methodoligies or whatever. We have code reuse! Productivity has obviously improved, it's just that there's also an arms race between software products in UI complexity, features, etc.

If you don't keep improving how efficiently you can ship value, your work will indeed be devalued. It could be that the economics shift such that pretty much all programming work gets paid less, it could be that if you're good and diligent you do even better than before. I don't know.

What I do know is that whichever way the economics shake out, it's morally neutral. It sounds like the author of this post leans into a labor theory of value, and if you buy into that, well...You end up with some pretty confused and contradictory ideas. They position software as a "craft" that's valuable in itself. It's nonsense. People have shit to do and things they want. It's up to us to make ourselves useful. This isn't performance art.

This has been discussed to death but to reiterate here: people have been much too polite about Kramnick's nonsense. Danya and Hikaru are (were) probably the two people in the world whose bullet play is least suspicious. Cheating isn't very powerful in short time controls, and they have streamed thousands of games playing them.

Kramnick's bullshit never made any damn sense at all.

In Europe there are legitimate and extremely established services that require you to input your bank login details into something other than your bank's website. It's madness.

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

It's interesting that there's still such a market for this sort of take.

> In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text."

What does this even mean? Let's veto the word "reasoning" here and reflect.

The LLM produces a series of outputs. Each output changes the likelihood of the next output. So it's transitioning in a very large state space.

Assume there exists some states that the activations could be in that would cause the correct output to be generated. Assume also that there is some possible path of text connecting the original input to such a success state.

The reinforcement learning objective reinforces pathways that were successful during training. If there's some intermediate calculation to do or 'inference' that could be drawn, writing out a new text that makes that explicit might be a useful step. The reinforcement learning objective is supposed to encourage the model to learn such patterns.

So what does "sophisticated simulators of reasoning-like text" even mean here? The mechanism that the model uses to transition towards the answer is to generate intermediate text. What's the complaint here?

It makes the same sort of sense to talk about the model "reasoning" as it does to talk about AlphaZero "valuing material" or "fighting for the center". These are shorthands for describing patterns of behaviour, but of course the model doesn't "value" anything in a strictly human way. The chess engine usually doesn't see a full line to victory, but in the games it's played, paths which transition through states with material advantage are often good -- although it depends on other factors.

So of course the chain-of-thought transition process is brittle, and it's brittle in ways that don't match human mistakes. What does it prove that there are counter-examples with irrelevant text interposed that cause the model to produce the wrong output? It shows nothing --- it's a probabilistic process. Of course some different inputs lead to different paths being taken, which may be less successful.

Like a backup generator for inputs. Makes sense.

LLM agents are very hard to talk about because they're not any one thing. Your action-space in what you say and what approach you take varies enormously and we have very little body of common knowledge about what other people are doing and how they're doing it. Then the agent changes underneath you or you tweak your prompt and it's different again.

In my last few sessions I saw the efficacy of Claude Code plummet on the problem I was working on. I have no idea whether it was just the particular task, a modelling change, or changes I made to the prompt. But suddenly it was glazing every message ("you're absolutely right"), confidently telling me up is down (saying things like "tests now pass" when they completely didn't), it even cheerfully suggested "rm db.sqlite", which would have set me back a fair bit if I said yes.

The fact that the LLM agent can churn out a lot of stuff quickly greatly increases 'skill expression' though. The sharper your insight about the task, the more you can direct it to do something specific.

For instance, most debugging is basically a binary search across the set of processes being conducted. However, the tricky thing is that the optimal search procedure is going to be weighted by the probability of the problem occurring at the different steps, and the expense of conducting different probes.

A common trap when debugging is to take an overly greedy approach. Due to the availability heuristic, our hypotheses about the problem are often too specific. And the more specific the hypothesis, the easier it is to think of a probe that would eliminate it. If you keep doing this you're basically playing Guess Who by asking "Is it Paul? Is it Anne?" etc, instead of "Is the person a man? Does the person have facial hair? etc"

I find LLM agents extremely helpful at forming efficient probes of parts of the stack I'm less fluent in. If I need to know whether the service is able to contact the database, asking the LLM agent to write out the necessary cloud commands is much faster than getting that from the docs. It's also much faster at writing specific tests than I would be. This means I can much more neutrally think about how to bisect the space, which makes debugging time more uniform, which in itself is a significant net win.

I also find LLM agents to be good at the 'eat your vegetables' stuff -- the things I know I should do but would economise on to save time. Populate the tests with more cases, write more tests in general, write more docs as I go, add more output to the scripts, etc.

Homelessness is a somewhat broad category though. There's lots of people couch-surfing between friends and their car. They're also in a very different position from people who are sleeping rough.

Carville (DNC strategist) is advocating a "play dead" strategy. Let Trump implode so that he owns the inevitable failure. His base will desperately want to blame the left for not letting the policies work as intended. The less the Democrats do, the harder that is. I think a lot of Democrat politicians are going this way, and it's why Schumer rolled over on the budget.

Part of the logic here is that Trump is indeed different from other authoritarians. He's even less competent. He's blowing all his political capital on imploding the economy. He also can't understand the legal battles, so when Stephen Miller tells him they won the Supreme Court case 9-0, he believes him. This seems to have been a big wake-up call to Gorsuch, Coney-Barrett and Kavanaugh. The administration has shown its hand much too quickly, before it fully consolidated its power.

What the Democrats should be doing already is campaigning more. Run ads that are literally just Trump quotes. Show people Trump calling January the "Trump economy" before inauguration, then calling April the "Biden economy" now that he's crashed it. If Trump polls low enough, more senators will jump ship, and impeachment could be possible.

It wasn't even so much the blunders as the strategic decisions I think. Like, a blunder isn't in itself "baffling".

You could do something like `value = table.get(some_function(), MISSING_VALUE)` and then have the conditional. But let's say for the sake of argument, yeah you need to assign the value up-front.

Let's say you're looking at some code like this:

    if value in table:
        ...

If you need to change this so that it's `some_function(value)`, you're not going to miss the fact that you have a decision to make here: you can either assign a new variable, or you can call the function twice, or you can use the '.get()' approach.

If you instead have:

    try:
        return table[value]
    except KeyError:
        ...

You now have to consciously avoid writing the incorrect code `try: return table[some_function(value)]`. It's very easy to change the code in the 'try' block so that you introduce an unintended way to end up in the 'except' block.

I still think using the exception mechanism for things that could just be conditionals is bad. I elaborated on this to a sibling comment.

Yes there's been various recommendations of this over the years and I think it's really bad.

Using try/except for conditional logic gives the developer a spurious choice between two different syntaxes to express the same thing. The reader is then asked to read try/except blocks as meaning two different things: either ordinary expected branching in the function, or handling exceptions.

I think it's definitely better if we just use conditionals to express conditionals, and try/except for errors, like every other language does. Here's some examples of where this duplication of syntax causes problems.

* Exceptions are often not designed to match the interface well enough to make this convenient. For instance, 'x in y' works for both mapping types and lists, but only mapping types will raise a `KeyError`. If your function is expected to take any iterable, the correct catching code will be `except (KeyError, IndexError)`. There's all sorts of these opportunities to be wrong. When people write exceptions, they want to make them specific, and they're not necessarily thinking about them as an interface to conveniently check preconditions. * Exceptions are not a type-checked part of the interface. If you catch `(KeyError, IndexError)` for a variable that's just a dictionary, no type checker (or even linter?) is going to tell you that the `IndexError` is impossible, and you only need to catch `KeyError`. Similarly, if you catch the wrong error, or your class raises an error that doesn't inherit from the class that your calling code expects it to, you won't get any type errors or other linting. It's totally on you to maintain this. * Exceptions are often poorly documented, and change more frequently than other parts of the interface. A third-party library won't necessarily consider it a breaking change to raise an error on a new condition with an existing error type, but if you're conditioning on that error in a try/except, this could be a breaking change for you. * The base exceptions are very general, and catching them in code that should be a conditional runs a significant risk of catching an error by mistake. Consider code like this:

    try:
        value = my_dict[some_function()]
    except KeyError:
        ...

This code is very seriously incorrect: you have no way of knowing whether 'some_function()' contains a bug that raises a KeyError. It's often very annoying to debug this sort of thing.

Because you must never ever ever call a function inside your conditional try block, you're using a syntax that doesn't compose properly with the rest of the language. So you can either rewrite it to something like this:

    value = some_function()
    try:
        return my_dict[value]
    except KeyError:
        ...

Or you can use the conditional version (`if my_dict[some_function()]`) just for these sorts of use-cases. But now you have both versions in your codebase, and you have to think about why this one is correct here and not the other.

The fundamental thing here is that 'try/except' is a "come from": whether you enter the 'except' block depends on which situations the function (or, gulp, functions) you're calling raise that error. The decision isn't local to the code you're looking at. In contrast, if you write a conditional, you have some local value and you're going to branch based on its truthiness or some property of it. We should only be using the 'try/except' mechanism when we _need_ its vagueness --- when we need to say "I don't know or can't check exactly what could lead to this". If we have a choice to tighten the control flow of the program we should.

And what do you buy for all this spurious decision making and the very high risk of very serious bugs anyway? Why should Python do this differently from every other language? I don't see any benefits in that article linked, and I've never seen any in other discussions of this topic.

The examples in the article recommend doing the right things, but I think the initial discussion of the LBYL vs EAFP discussion could be better.

The article says basically "LBYL is bad", but this isn't a good description of what the author does in the later examples. Following "EAFP" without exception is also really bad.

The simple policy is that you should use "LBYL" when you're dealing with local state that can't change out from under you, especially when it's just properties of your local variables, but you need to use "EAFP" for anything that deals with remote state. Function calls can go either way, depending.

Blanket advice against LBYL leads people to use try/except blocks to express basic conditionals. If you want to ask something about your local variables, just ask it in a conditional --- don't make the reader infer "ah, these are the situations which will raise the error, and therefore we'll go into this block if the values are this way". If you want to do something when a key is missing from a dictionary, don't write:

    try:
        value = table[key]
    except KeyError:
        return default_value

Just write:

    if key not in table:
        return default_value
    value = table[key]

(If you really need to avoid two lookups for efficiency, you would do something like value = table.get(key, MISSING_VALUE) and then check 'if value is missing_value'. But this optimisation will seldom be necessary.)

The example the author gives about interacting with the file system is a good example of where indeed you really should use EAFP. The file system is remote state that your function does not own. Similarly if you're doing database operations, calling a remote API...lots of things.

There's various middle ground when you're calling functions. Often you don't want to worry about whether that function is going to be interacting with remote state, and you want to treat it totally as a black box, so you just use EAFP. But if the function documents really clear pre-conditions of your variables you can just go ahead and check, it's better to do that.

Maybe "trust" in a purely technical sense, but we live in a society. If a vendor has a part of their contract that would put them under penalty of fraud, customers don't usually worry whether they can independently verify it's true.

For instance, if you write in your agreements that data goes into this bit of open-source code and leaves it and goes nowhere else, and you list out some processes you take to ensure this is true, and the whole thing is an outright lie, that's really bad for you in ways that are clear and enforceable. It's also hard to hide from your own employees, and people come and go. You're unlikely to make that claim unless it's true.

The gold standard they're comparing against was done by humans though. And a task-specific model trained on that data will be better at that task than GPT-4.

What's definitely true is that getting decent data often takes some care, especially in how you define the task. And mechanical turk is often especially tricky to use well.

You probably know this but you definitely don't have to run into that problem. In practice most people who use one component to produce features for another will take care to ensure errors are present in the pipeline. So the conceptually simple (but operationally annoying) way to do this is to train your POS tagger or whatever on multiple folds, and predict the missing fold. This is known as "jack-knife training" in the literature.

In spaCy what we do is just train the components in sequence. So everything is trained at the same time, and in the early iterations the model is seeing samples with errors. I've always found this to be good enough.

There's definitely active research on it. But here's the basic recipe that gets state-of-the-art accuracy on most tasks. It's been around for 5 years or so now, which is why I said "just".

You take an encoder transformer architecture like BERT and train it on a language modelling objective. Then you discard the part of the network that does the token prediction, and stick some randomly initialised network that does a specific task on. This is generally kept minimal. For classification tasks, often people just connect a linear layer to the vector that represents a dummy sentinel token after the sequence.

The general goal is to exploit the representations learned during the language modelling task, so that the network starts out knowing general grammatical structure of the language, multi-word expressions, can distinguish word senses from each other, etc. Then it needs far fewer examples to learn a specific task. There's definitely transfer learning going on: if you initialise the network randomly instead of via the LM objective, results are very much worse.

This is a good blog post from when the recipe was still fairly new: https://www.ruder.io/nlp-imagenet/

(Author here)

About the speed, the idea is that if you break down the task, you can very often use much smaller models for the component tasks. LLMs are approaching prediction tasks under an extremely difficult constraint: they don't get to see many labelled examples. If you relax that constraint and just use transfer-learning, you can get better accuracy with much smaller models. The transfer-learning pipeline can also be arranged so that you encode the text into vectors once, and you apply multiple little task networks over the shared representation. spaCy supports this for instance, and it's easy to do when working directly with the networks in PyTorch etc.

This is where the terminology being used to discuss LLMs today is a touch awkward and imprecise.

There's a key distinction between smaller models trained with transfer-learning, and just fine-tuning a smaller LLM and still using in-context learning.

Transfer learning means you're training an output network specifically for the task you're doing. So like, if you're doing classification, you output a vector with one element per class, apply a softmax transformation, and train on a negative log likelihood objective. This is direct and effective.

Fine-tuning a smaller LLM so that it's still learning to do text generation, but it's better at the kinds of tasks you want to do, is a much more mixed experience. The text generation is still really difficult, and it's really difficult to learn to follow instructions. So all of this still really favours size.

Nothing's to stop us, and in fact we can do that now! This is basically what the post advocates for: replacing the LLM calls for task-specific things with smaller models. They just don't need to be LLMs.

(Author here)

It's true that getting something going end-to-end is more important than being perfectionist about individual steps -- that's a good practical perspective. We hope good evaluation won't be such an enormous effort. Most of what we're trying to do at Explosion can be summarised as trying to make the right thing easy. Our annotation tool Prodigy is designed to scale down to smaller use-cases for instance ( https://prodigy.ai ). I admit it's still effort though, and depending on the task, may indeed still take expertise.

I think the 'industrial' or assembly-line analogy is probably good, and I see what you mean about the alternative system design. Thanks for explaining the other approach.

I'll put it this way. If you want to integrate ML into a product, or even a system with lots of internal users, you end up increasingly towards the 'factory' approach.

I'm making the stronger claim that fine-tuned BERT is almost always better than ICL GPT4. That is, you beat ICL GPT4 with a model of less than 1b parameters. These are the results in the literature.

Fine-tuning GPT3 is actually a strange middle-ground: you're constrained to solve the tasks via prompting, rather than attaching output layers and doing gradient descent over a loss function that directly matches the task. So fine-tuning LLMs for these classification tasks is indeed often worse than just using the larger model.

The other thing is, as you scale up the model the ratio of output weights to the whole model weights gets quite out of balance. This introduces some technical challenges, and so it's kind of tough to make it all work with medium size LLMs of around 7b parameters or so. In contrast the process of fine-tuning BERT-sized stuff is really well explored, and it's easy to find more or less push-button solutions.

I know what you mean about the domain knowledge, and it's a thing that's a bit different from the previous situation with the feature engineering. The problem with feature engineering for linear models was you really had to understand the domain _and_ the ML.

I do think there's a similarity in how creative you need to be though. It means that applying LLMs to new problems isn't as straight-forward as people make it seem at first, and isn't necessarily reliable. In contrast, labelling data is something that has a much smoother effort to reward curve for most problems. The experience of labelling the data, training a model and getting it hosted isn't as seamless as it could be -- we're working on that.

I do think classification is pretty fundamental though. The way I see it is, model outputs can be either human-facing, machine facing, or both. If you're going to feed the output into another system, that system wants the data to obey some limited schema, so that you can run logic based on it. For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task. You might want to also attach text in the alert, so that's a human-facing part.

I agree that there's lots of ways that LLMs can be used iteratively, allowing more trade-off of computational cost for accuracy. I just think in a lot of cases, the best way to exploit that is to trade towards as much accuracy as you can get, and use that to create training samples. You can then manually correct the training samples as well --- if they're mostly correct, reviewing them is pretty quick. You can then train and evaluate a smaller model.

(Original author of spaCy and Explosion CTO here)

Okay so, first some terminology. LLMs can mean a bunch of different things, people call models the size of BERT LLMs sometimes. So let's talk specifically about in-context learning (ICL) with either zero or a few examples. So we'll say LLM ICL, and contrast that with techniques where you annotate enough data to train with, which might only be something like 10-40 hours of annotation. The something you do with that data is probably training a task-specific classification model initialised with weights from a language modelling objective. This is sometimes called "fine-tuning", but fine-tuning can also mean taking an LLM and adapting its ICL. So we'll just call it "training", and the fact you use transfer learning like BERT or even word vectors is just tactics.

Okay. So, here's something that might surprise you: ICL actually sucks at most predictive tasks currently. Let's take NER. Performance on NER on some datasets is _below 2003_. Here's some recent discussion: https://twitter.com/mayhewsw/status/1700139745769046409

The discussion focusses on how bad the CoNLL 2003 dataset is, and indeed it's a crap dataset. But experiments have also been done on other datasets, e.g. check out the comparison of ICL and training in this paper from Microsoft: https://universal-ner.github.io/ . When GPT4 is used this one paper reports it slightly better on some tasks: https://arxiv.org/abs/2308.10092 . Frustratingly they don't do enough GPT4 experiments. This other paper also does a huge number of experiments, but not with GPT4: https://arxiv.org/pdf/2303.10420.pdf

The findings across the literature are really clear. ICL is generally much worse than training a model in accuracy, and you generally don't need much training data to surpass ICL in accuracy.

For tasks like text classification, ICL sometimes does okay. But you need to pick the problem characteristics carefully. Most text classification tasks people actually want to do have something like 20 labels, the texts are kind of long, and the labels don't capture the category especially well. Applying ICL to such tasks is very non-trivial. Your prompt balloons up if you have lots of classes to predict between, and providing the examples is hard if your texts are even a few hundred words.

Let's say you want to do something ultra simple: classify articles into categories for some news site or blog. This is the type of problem text classifiers have been eating for breakfast for 20 years. This is not a difficult problem -- a unigram bag of words does fine, and the method of estimating the weights can be almost anything, like just averaged perceptron will be totally okay.

But how will an LLM be able to do this? Probably your topic categories include several different types of article under them. If you know what those types of article are you can separate them out and make sure they're represented in the prompt. But now we're back at needing a tonne of domain knowledge about your problem -- that's like having to write custom features to make your model work. We all switched to deep learning so we wouldn't have to do that.

LLMs build a much more sophisticated representation of the meaning of the data to classify. But then you give them very few examples of the problem. So they can only build a shallow function from this deep representation. If you have lots of examples, you can learn a complex function from shallower features. For a great many classification tasks, this is better. The rest of your system usually needs the classification module to have some sort of consistent behaviours anyway. To do that, you basically have to make an annotation manual, and then you want to annotate evaluation documents. Once you're there the effort to make training data and train a model is minimal.

The other elephant in the room is the expense of the LLM solutions. The papers are missing results on GPT4 not because they're lazy, but because it's so expensive to use GPT4 as a classification solution that they want to get the rest of their results out the door.

The world cannot migrate all its current NLP models for text classification and NER to ICL. There are nowhere near enough GPUs in the world for that to happen. And I don't know about you, but I expect the number of text classification and NER models to grow, not shrink. So, the idea that we'll stop training small models for these tasks is just implausible. The OpenAI models that support batching are almost viable for prediction, but models like GPT4 don't support it (perhaps due to the mixture of experts?), so it's super slow.

The other thing is, many aspects of language that are useful as annotations are consistent linguistic features. The English language codes for proper names and numeric entities. They behave differently in the grammar. So some sort of named entity annotation can be done once, and then the model trained and reused. This is what spaCy does. We do this for a variety of other useful annotations across languages. We actually need to do much more: we need to collect new annotations for these models to keep them up to date, and we need to do this for more tasks, such as semantic role labelling. But it's definitely a good way to reuse work. We can do this annotation once, train the models, and users can reuse the models.

The strength of ICL is that you can get started very easily, without doing the work of annotation and training. There's lots of research on making ICL few-shot learning less bad on arbitrary text classification, NER and other tasks. We're working hard to take these results from the literature and build best-practice prompts and parsers you can use as a drop-in annotation module in spaCy: https://github.com/explosion/spacy-llm . Our annotation tool Prodigy also supports initializing the annotations from an LLM, and just correcting the output: https://prodigy.ai . The idea is to let you start with an LLM, and then transition to a model you train yourself, which can be run much faster.

The idea is to hash the string into a 64-bit key. You can store the string in a value, or you can have a separate vector and make the value a struct that has the key and the value.

The chance of colliding on the 64-bit space is low if the hash distributes evenly, so you just yolo it.

If you have very large dicts, you might find this hash table I wrote for spaCy helpful: https://github.com/explosion/preshed . You need to key the data with 64-bit keys. We use this wrapper around murmurhash for it: https://github.com/explosion/murmurhash

There's no docs so obviously this might not be for you. But the software does work, and is efficient. It's been executed many many millions of times now.