capnrefsmmat

But it isn't neutral average language; it's extremely idiosyncratic. GPT-4o used present participles something like 5 times more often than the average human, for instance.

Though there evidently was an attempt to make GPT-5 sound more neutral, but in a literal sense: the emotional content of its text was flattened out. Suddenly it wasn't nearly as sycophantic or enthusiastic, and wrote much more dry, neutral text.

I think it's more accurate to say the text is not regressing to a mean, but to an artificially constructed style defined by the training process. And the AI companies have only partial control over that style, so weird things emerge from it.

The "regression to the mean" or "it's just the training data" hypothesis doesn't work. Pretrained models do not have these weird writing styles and sound much more human. It's only after instruction tuning -- SFT, DPO, etc. -- that LLMs develop weird writing styles.

That is, the original training on massive text corpora seems to produce an average of human writing, but that average looks pretty average. It doesn't feel like LLM slop. But the instruction tuning on specific tasks (answer this prompt, summarize this text, do this math problem), which turns text predictors into chatbots, seems to create strange style features as a side effect. And because that tuning step depends on company-specific training data (rather than just a crawl of the entire Internet), each company's LLM has a different style.

It's not that difficult to measure this. Linguists have had measures of style variation for a long time, though NLP and AI researchers seem to have forgotten. We've generated parallel human and AI corpora to measure it, and can show specifically which writing features change: https://arxiv.org/abs/2410.16107

Currently we're working on tweaking the instruction tuning process to see if we can pin down which steps and what types of tasks tend to produce the weird style features.

This theory doesn't hold up when you use base models (i.e., models with only pretraining, no RLHF or instruction tuning). They sound much more human. I've tested pairs (like Llama 3 base vs. instruct), and the base model can match human style much better than the instruct version can: https://www.pnas.org/doi/10.1073/pnas.2422455122

I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)

This is the most memorable abstract I've ever read.

Smith, Gordon C. S., and Jill P. Pell. 2003. “Parachute Use to Prevent Death and Major Trauma Related to Gravitational Challenge: Systematic Review of Randomised Controlled Trials.” BMJ 327 (7429): 1459–61. https://doi.org/10.1136/bmj.327.7429.1459.

Objectives: To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.

Design: Systematic review of randomised controlled trials.

Data sources: Medline, Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists.

Study selection: Studies showing the effects of using a parachute during free fall.

Main outcome measure: Death or major trauma, defined as an injury severity score > 15.

Results: We were unable to identify any randomised controlled trials of parachute intervention.

Conclusions: As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.

It's not the training data. When we've tested the writing style of base models that have not gone through instruction tuning, they're much more human-like (https://arxiv.org/abs/2410.16107). The style shift seems to come from something in the instruction-tuning process, so our current research problem is figuring out what in the process is doing it.

More recent surveys have not contradicted the trend. For instance, in 2015,

> On average, students spent about 14 HPW preparing for class, 6 HPW working for pay, 12 HPW relaxing and socializing, 7 HPW participating in co-curricular activities and community service, 2 HPW caring for dependents, and 3 HPW commuting to campus. These activities accounted for a total weekly average of 45 HPW.

https://doi.org/10.1007/s11162-018-9497-z

I've looked for other data and have not found anything to indicate the trend has reversed.

> He seems quite out of touch with the level of economic hardship involved in being a student; a huge number of students already work a couple of jobs and are navigating small businesses, especially here in the UK where he apparently now lives. They aren't going to have enough "free time" to play with a "project" even if a bit more is made available by loosening the curriculum.

Empirically, student free time grew for much of the 20th century, and students spend much less time on school work than they used to. The decline is not explained by students spending more time on work outside of school; much of it has become leisure time.

https://doi.org/10.1162/REST_a_00093 https://doi.org/10.1177/016146811011200908

Now perhaps there has been a sharp change in the years since those two studies (there certainly has been a sharp change in UK tuition costs), but the point is that our intuitions about how hard students must have to work these days are usually wrong.

> Well, what do you expect? LLMs are trained on blithering, mostly from web sites. So you get blithering out.

I realize this isn't entirely serious, but I can't resist pointing out that this doesn't seem to be a good explanation for why LLMs write the way they do. When we've experimented with LLM writing style on open-weights models where you can get a base model (pretraining on text only) and an instruction-tuned variant (pretraining + post-training with RLHF and whatever other human-evaluated tasks), it's the instruction-tuned variant that shows the weird writing quirks. That is, the writing style is not because of the training texts, but because of whatever tasks the LLM companies do in instruction tuning. https://arxiv.org/abs/2410.16107

I'd speculate that this is partly impressed human preferences (the human raters unintentionally reward a particular writing style) and partly because of the chosen tasks: they're training the LLM to be good at, say, summarizing text, so it develops a style that's good at being informationally dense.

At any rate I've seen this same phenomenon with Llama and Gemma, and will be trying soon with Qwen. Unfortunately none of the commercial models lets you access the base model, as far as I know.

Being invited to conference talks around the world is a completely normal part of being an active researcher in almost any academic field, so it doesn't register as pompous to other academics.

From your link,

> TerraPower must still complete construction, submit an operating license application, and satisfy all applicable safety and regulatory requirements before loading fuel and beginning operations.

Discussion is included in the Dangerous Goods Panel report, agenda item 4.3 (pages 39-41) and Appendix E (beginning page 89). https://www.icao.int/sites/default/files/DangerousGoods/DGP%...

Paragraph 4.3.3:

> While data indicated that portable electronic devices were more often the cause of fire in aircraft cabins than power banks were, the latter were a significant concern due to their increased use and a prevalence of lower-quality products with defects or vulnerabilities that were more likely to lead to thermal events. Power banks were also not offered the same level of protection that batteries installed in portable electronic devices were provided. The amendments therefore focused on power banks.

Several reasons:

1. The post mainly reiterates a single idea (Capsicum enumerates what the process can do, seccomp provides a configurable filter) in many different ways. There is not much actual depth, code samples notwithstanding. Nothing on why different designs were chosen, how easy each is to use, outcomes besides the Chrome example, etc.

2. There are a lot of AI writing tells, like staccato sentences, parallelism ("Same browser. Same threat model. Same problem."), pointless summary tables, "it's not X, it's Y" contradiction ("This is not a bug. It is the original Unix security model"), etc.

3. The author has roughly a blog post a day, all with similar style and on widely varied topics, and in the same writing style. Unless the author has deep expertise on a remarkably wide range of topics and spends all their time writing, these can't reflect deep insight or experience, but minimal editing of AI output.

So yes, it's pretty sloppy.

Probably. One common feature of LLM output is grammatical features that indicate information density, like nominalizations, longer words, participial clauses, and so on. Perhaps training tasks that involve asking the LLMs for concise explanations or summaries encourage the use of these features to give denser answers.

Thanks for the links. You may be interested in the other LLM writing style studies I've been collecting: https://www.refsmmat.com/notebooks/llm-style.html

I've heard the Kenya and Nigeria story, but has anyone backed it up with quantitative evidence that the vocabulary LLMs overuse coincides with the vocabulary that is more common in Kenyan and Nigerian English than in American English?

I work on research studying LLM writing styles, so I am going to have to steal this. I've seen plenty of lists of LLM style features, but this is the first one I noticed that mentions "tapestry", which we found is GPT-4o's second-most-overused word (after "camaraderie", for some reason).[1] We used a set of grammatical features in our initial style comparisons (like present participles, which GPT-4o loved so much that they were a pretty accurate classifier on their own), but it shouldn't be too hard to pattern-match some of these other features and quantify them.

If anyone who works on LLMs is reading, a question: When we've tried base models (no instruction tuning/RLHF, just text completion), they show far fewer stylistic anomalies like this. So it's not that the training data is weird. It's something in instruction-tuning that's doing it. Do you ask the human raters to evaluate style? Is there a rubric? Why is the instruction tuning pushing such a noticeable style shift?

[1] https://www.pnas.org/doi/10.1073/pnas.2422455122, preprint at https://arxiv.org/abs/2410.16107. Working on extending this to more recent models and other grammatical features now

No, that doesn't really work so well. A lot of the LLM style hallmarks are still present when you ask them to write in another style, so a good quantitative linguist can find them: https://hdsr.mitpress.mit.edu/pub/pyo0xs3k/release/2

That was with GPT4, but my own work with other LLMs show they have very distinctive styles even if you specifically prompt them with a chunk of human text to imitate. I think instruction-tuning with tasks like summarization predisposes them to certain grammatical structures, so their output is always more information-dense and formal than humans.

The first sentence is a reference to prior research work that has found those productivity gains, not a summary of the experiment conducted in this paper.

Most of the tedious formatting requirements do not match what the final typeset article looks like. The requirements are instead theoretically to benefit peer reviewers, e.g., by having double-spaced lines so they can write their comments on the paper copy that was mailed to them back when the submission guidelines were written in the 1950s.

The smarter journals have started accepting submissions in any format on the first round, and then only require enough formatting for the typesetters to do their job.

Outside of disciplines that use LaTeX, the ability of authors to do typesetting is pretty limited. And there are other typesetting requirements that no consumer tool makes particularly easy; for instance, due to funding requirements, many journals deposit biomedical papers with PubMed Central, which wants them in JATS XML. So publishers have to prepare a structured XML version of papers.

Accessibility in PDFs is also very difficult. I'm not sure any publishers are yet meeting PDF/UA-2 requirements for tagged PDFs, which include things like embedding MathML representations of all mathematics so screenreaders can parse the math. LaTeX only supports this experimentally, and few other tools support it at all.

It didn't "survey" devs. It paid them to complete real tasks while they were randomly assigned to use AI or not, and measured the actual time taken to complete the tasks vs. just the perception. It is much higher quality evidence than a convenience sample of developers who just report their perceptions.

Sure, if you're learning to write and want lots of examples of a particular style, LLMs can generate that for you. Just don't assume that is a normal writing style, or that it matches a particular genre (say, workplace communication, or academic writing, or whatever).

Our experience (https://arxiv.org/abs/2410.16107) is that LLMs like GPT-4o have a particular writing style, including both vocabulary and distinct grammatical features, regardless of the type of text they're prompted with. The style is informationally dense, features longer words, and favors certain grammatical structures (like participles; GPT-4o loooooves participles).

With Llama we're able to compare base and instruction-tuned models, and it's the instruction-tuned models that show the biggest differences. Evidently the AI companies are (deliberately or not) introducing particular writing styles with their instruction-tuning process. I'd like to get access to more base models to compare and figure out why.

I don't think the AI companies are systematically working to make their models sound more human. They're working to make them better at specific tasks, but the writing styles are, if anything, even more strange as they advance.

Comparing base and instruction-tuned models, the base models are vaguely human in style, while instruction-tuned models systematically prefer certain types of grammar and style features. (For example, GPT-4o loves participial clauses and nominalizations.) https://arxiv.org/abs/2410.16107

When I've looked at more recent models like o3, there are other style shifts. The newer OpenAI models increasingly use bold, bulleted lists, and headings -- much more than, say, GPT-3.5 did.

So you get what you optimize for. OpenAI wants short, punchy, bulleted answers that sound authoritative, and that's what they get. But that's not how humans write, and so it'll remain easy to spot AI writing.

Probably because the article uses the Unicode right single quotation mark instead of apostrophes, due to some automated smart-quote machinery. I'll have to adjust the tagger to handle those.

In our studies of ChatGPT's grammatical style (https://arxiv.org/abs/2410.16107), it really loves past and present participial phrases (2-5x more usage than humans). I didn't see any here in a glance through the lightfastness section, though I didn't try running the whole article through spaCy to check. In any case it doesn't trip my mental ChatGPT detector either; it reads more like classic SEO writing you'd see all over blogs in the 20-teens.

edit: yeah, ran it through our style feature tagger and nothing jumps out. Low rate of nominalizations (ChatGPT loves those), only a few present participles, "that" as subject at a usual rate, usual number of adverbs, etc. (See table 3 of the paper.) No contractions, which is unusual for normal human writing but common when assuming a more formal tone. I think the author has just affected a particular style, perhaps deliberately.

OpenAI is the custodian of the user data, so they are responsible. If you wanted the court (i.e., the plaintiffs) to find specific infringing chatters, first they'd have to get the data from OpenAI to find who it is -- which is exactly what they're trying to do, and why OpenAI is being told to preserve the data so they can review it.

If the output is interpreting sources rather than just regurgitating quotes from them, you need to exert judgment to verify they support its claims. When the LLM output is about some highly technical subject, it can require expert knowledge just to judge whether the source supports the claims.

Courts have always had the power to compel parties to a current case to preserve evidence. (For example, this was an issue in the Google monopoly case, since Google employees were using chats set to erase after 24 hours.) That becomes an issue in the discovery phase, well after the defendant has an opportunity to file a motion to dismiss. So a case with no specific allegation of wrongdoing would already be dismissed.

The power does not extend to any of your hypotheticals, which are not about active cases. Courts do not accept cases on the grounds that some bad thing might happen in the future; the plaintiff must show some concrete harm has already occurred. The only thing different here is how much potential evidence OpenAI has been asked to retain.

For introductory problems, the kind we use to get students to understand a concept for the first time, the AI would likely (nearly) nail it on the first try. They wouldn't have to fix any non-working code. And annotating the code likely doesn't serve the same pedagogical purpose as writing it yourself.

Students emerge from lectures with a bunch of vague, partly contradictory, partly incorrect ideas in their head. They generally aren't aware of this and think the lecture "made sense." Then they start the homework and find they must translate those vague ideas into extremely precise code so the computer can do it -- forcing them to realize they do not understand, and forcing them to make the vague understanding concrete.

If they ask an AI to write the code for them, they don't do that. Annotating has some value, but it does not give them the experience of seeing their vague understanding run headlong into reality.

I'd expect the result to be more like what happens when you show demonstrations to students in physics classes. The demonstration is supposed to illustrate some physics concept, but studies measuring whether that improves student understanding have found no effect: https://doi.org/10.1119/1.1707018

What works is asking students to make a prediction of the demonstration's results first, then show them. Then they realize whether their understanding is right or wrong, and can ask questions to correct it.

Post-hoc rationalizing an LLM's code is like post-hoc rationalizing a physics demo. It does not test the students' internal understanding in the same way as writing the code, or predicting the results of a demo.

> Well, if you’re a novice, don’t do that.

I agree, and it sounds like you're getting great results, but they're all going to do it. Ask anyone who grades their homework.

Heck, it's even common among expert users. Here's a study that interviewed scientists who use LLMs to assist with tasks in their research: https://doi.org/10.1145/3706598.3713668

Only a few interviewees said they read the code through to verify it does what they intend. The most common strategy was to just run the code and see if it appears to do the right thing, then declare victory. Scientific codebases rarely have unit tests, so this was purely a visual inspection of output, not any kind of verification.