How we measured AI writing across arXiv, and where the measurement breaks

Re:How we measured AI writing across arXiv, and where the me... by lingeringpine on Monday July 20, 2026 @01:40PM (#48982121)

I am not a native English speaker. This is not surprising to me. I think most of the papers we write would be flagged by AI detectors.

It is not because we ask LLM to write us a paper about X. It is because we are bad at writing in a scientific style, and american editors expect us to do it. With LLMs, we can write in basic sentences and tell the LLM the idea and it converts that to nice writing.

If you write each paragraph and have an LLM make that paragraph more scientific, it is entirely your paper, but it is flagged as LLM generated. If you have an LLM write the paper but speak good enough English, you can make it look human even though it is not human.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by dopamine_daddy on Monday July 20, 2026 @01:50PM (#48982265)

Yes that is exactly what I suspected. And I see absolutely nothing wrong with this.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by vidarh on Monday July 20, 2026 @02:05PM (#48982527)

My English is fine, and I did hand-write my MSc thesis in English, but scientific style writing is very tedious. Personally I don't think I'd even consider writing a paper "by hand" if I were to write another one.

There really is no point, as long as you verify the content matches your intent and edit out anything poorly written.

Frankly, I've read plenty of papers by native English-speakers over the years that'd strongly benefit from being rewritten by an LLM too...

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by bananaflag on Monday July 20, 2026 @01:48PM (#48982215)

> It is because we are bad at writing in a scientific style, and american editors expect us to do it. With LLMs, we can write in basic sentences and tell the LLM the idea and it converts that to nice writing.

This is a sin.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by exe34 on Monday July 20, 2026 @02:05PM (#48982510)

I feel like they should release the original before the llm chewed on it.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by JadeNB on Monday July 20, 2026 @02:05PM (#48982509)

Which part is a sin? Using LLMs to deal with a lack of English-language fluency? I am a scientist (actually a mathematician, if it matters), and, if that's the way to deal with the practical hegemony of English in the scientific literature, then I have no problem with it. Rather that than people with important ideas can't get them before the scientific community. As long as the authors personally check and stand behind the scientific content of their papers, what do I care how the exposition was produced, especially if the role of LLMs is properly disclosed?

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by never_inline on Tuesday July 21, 2026 @12:47AM (#48988155)

https://lists.isocpp.org/std-proposals/att-0486/Reply_to_Zer...

Here's a paper by a non native English speaker. The lack of the formal style doesn't cause any issue. Preciseness is what matters.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by rcxdude on Tuesday July 21, 2026 @04:18AM (#48989526)

To your tastes maybe. Will it get through the editor's desk at a scientific journal as easily?

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by JadeNB on Wednesday July 22, 2026 @08:45AM (#49005923)

It seems to me like we're saying the same thing. I'm saying that there is a practical bias, as in, it's there in the real world whether we like it or not, that can prevent some papers from appearing just because of limitations on English-language fluency, not because of any lack of scientific content. One solution is to get rid of it, but that's a long-term project. If we have an immediate remedy that works, if the authors still take responsibility for the scientific content, and if the use of the tool that provides that remedy is responsibly disclosed, then it seems like a win all around, except perhaps that it makes the need for the longer-term solution less obvious.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by jazzyjackson on Tuesday July 21, 2026 @10:10AM (#48992600)

It would be nice if scientists could write and publish in their first language. If a machine translation brings it to a wider audience fine, but the lack of disclosure of who authored the paper is a problem.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by JadeNB on Wednesday July 22, 2026 @08:47AM (#49005940)

I'd way rather read a translation in which the author themselves has played a part, and at least nominally signed off on it, than one done on the fly with even less than usual guarantee of its correctness.

(I agree that authors should be able to publish in their first language, but I also note that the historical practicality has been that publication in non-dominant languages has resulted more in silos than in broadening access.)

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by zfnmxt on Monday July 20, 2026 @01:57PM (#48982356)

> If you write each paragraph and have an LLM make that paragraph more scientific, it is entirely your paper

"More scientific" is not some merely stylistic thing that faithfully preserves the original meaning of what you wrote. The precise details of each paragraph matters a lot in terms of what and how it communicates. The fact that these details do matter means that, according to my accounting, it is not entirely your paper.

Also, I find striving for "scientific" to be a pretty undesirable thing. Why should papers read like that? What is the benefit? The best papers (in terms of their writing and communication) are unpretentious and conversational. I'm pretty sure I'd prefer your "basic sentences", especially if they were wholly yours. (I understand that there are also external forces at play here as you mentioned.)

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by smcg on Monday July 20, 2026 @02:01PM (#48982439)

You might prefer that, but journal editors might not.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by zfnmxt on Monday July 20, 2026 @02:06PM (#48982536)

I've never had any trouble publishing in a more conversational style and have read plenty of papers that do the same. Of course, this varies by field and venue and format (journal vs. conference) and even reviewer and it's difficult to understand what the true constraints are and what the state of the publishing system is in this regard.

But I suspect that a lot of academic's feelings about it are informed by what others have told them and how they've been trained, rather than by what's actually permissible in the publishing system.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by simonreiff on Monday July 20, 2026 @06:08PM (#48985488)

Being charitable, "more scientific" means more rigorous, reproducible, reliable based on the consensus of that scientific community (if any consensus exists, which it usually does in math and hard sciences), transparent in methodology, and candid about authorities cited and limitations to the experimental approach. Science is not a style guide; the purpose of a research publication is to impart a result to experts in the field and make claims that can be rigorously scrutinized. The best way to make a claim credibly is to demonstrate full fluency with the tools and techniques of the academic community in applying them thoughtfully to the chosen problem. But the result of doing that often looks downright impenetrable to the general public; see, e.g., Niven 1947 (1-page proof that pi is irrational). It's not the point of science to make it accessible to everyone. The point is to convince experts in the field of the correctness of the claim. The audiences are often totally different, and the tone and qualities of the writings likewise must vary.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by bjourne on Monday July 20, 2026 @01:46PM (#48982192)

There is a fine line between using an LLM to clean up grammar and spelling and using it for text generation.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by robotsliketea on Monday July 20, 2026 @09:58PM (#48987195)

It seems that LLMs often insert new ideas even when asked only to polish writing without changing it: https://bsky.app/profile/tomerullman.bsky.social/post/3mq33c...

I don't think this is surprising. Good technical writing is very precise. If you're starting from non-technical writing, I suspect that in most cases you can't make it "sound scientific" without adding new claims or changing the meaning. (Maybe you are being more careful, but this is something that worries me in general.)

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by aionwikipedia on Tuesday July 21, 2026 @12:57AM (#48988203)

most papers like this are flagged by AI detectors because, by your own admission, AI was used to generate the final output. what exactly did you expect here?

[ Parent | on HN ]