Some words/phrases that, by default, it overuses: "dive into", "delve into", "the world of", and others.
You correct it with instructions, but it will then find synonyms so there is also a structural pattern to the output that it favors by default. For example, if we tell it "Don't start your writing with 'dive into'", it will just switch to "delve into" or another synonym.
Yes, all of this can be corrected if you put enough effort into the prompt and enough iterations to fix all of these tells.
LLMs can radically change their style, you just have to specify what style you want. I mean, if you prompt it to "write in the style of an angry Charles Bukowski" you'll stop seeing those patterns you're used to.
In my team for a while we had a bot generating meeting notes "in the style of a bored teenager", and (besides being hilarious) the results were very unlike typical AI "delvish".
Aside: not about you specifically, but I feel like complaints on HN about using LLMs often boil down to somebody saying "it doesn't do X", where X is a thing they didn't ask the the model to do. E.g. a thread about "I asked for a Sherlock Holmes story but the output wasn't narrated by Watson" was one that stuck in my mind. You wouldn't think engineers would make mistakes like that, but I guess people haven't really sussed out how to think about LLMs yet.
Anyway for problems like what you described, one has to be wary about expecting the LLM to follow unstated requirements. I mean, if you just tell it not to say "dive into" and it doesn't, then it's done everything it was asked, after all.
You'd have to come up with a pretty exhaustive list of tells. Even sentence structure and mood is sometimes enough, not just the obvious words.
Also, wouldn't angry Charles Bukowski just be ... Charles Bukowski?
That is true, but more importantly, are those patterns sufficient to distinguish between AI-generated content from human-generated content? Humans express themselves very differently by region and country ( e.g. "do the needful" in not common in the midwest, "orthogonal" and "order of magnitude" are used more on HN than most other places). Outside of watermaking, detecting AI-generated text is with an acceptably small false-positive error rate is nearly impossible.
Maybe a database could be built with “tells” organized by model.
Automated by the LLMs themselves.
Regular ol tests would do
Thinking about it a bit more, the tells that work might depend on the usage of other specific prompts.
I didn't say they know their own tells. I said they naturally output them for you. Maybe the obvious is so obvious I don't need to comment on it. Meaning this whole "tells analysis" would necessarily rely on synthetic data sets.
AI detectors generally can take advantage of this and look for abnormal patterns in frequencies of specific words, phrases, or even specific grammatical constructs because the LLM -- by default -- is biased that way.
I'm not saying this is easy and certainly, LLMs can be tuned in many ways via instructions, context, and fine-tuning to mask this.
All that aside, most models have had a fairly distinctive writing style, particularly when fed no or the same system prompt every time. If o1-Pro blends in more with human writing that's certainly... interesting.
[0] https://openai.com/index/new-ai-classifier-for-indicating-ai...
Damnit. It's too good. It just saved me ~6 hours in drafting a complicated and bespoke legal document. Before you ask: I know what I'm doing, and it did a better job in five minutes than I could have done over those six hours. Homework is over. Journalism is over. A large slice of the legal profession is over. For real this time.
It just replaces human slop with automated slop. It doesn't automate finding hidden things out just yet, just automates blogspam.
It's actually surprising how many articles on 'respected' news websites have typos. You'd think there would be automated spellcheckers and at least one 'peer review' (probably too much to ask an actual editor to review the article these days...).
AI can handle that sort of writing just fine, readers won't care about the formulaic writing style.
It's actually surprising how many articles on 'respected' news websites have typos.
> I know what I'm doing
Is exactly the key element in being able to use spicy autocomplete. If you don't know what you're doing, it's going to bite you and you won't know it until it's too late. "GPT messed up the contract" is not an argument I would envy anyone presenting in court or to their employer. :)
(I say this mostly from using tools like copilot)
Seems like lawyers could do more faster because they know what they are doing. Experts dont get replaced, they get tools to amplify and extend their expertise
Edit> Its good. Thanks again for ur review.
Just like when software that was coming out, it may have ended jobs.
But it also helped get things done that wouldn't otherwise, or as much.
In this case, equipping a capable lawyer to be 20x is more like an iron man suit, which is OK. If you can get more done, wit less effort, you are still critical to what's needed.
The e-mail correspondence goes like this: "Hello Professor, I'd like to meet to discuss my failing grade. I didn't know that using ChatGPT was bad, can I have some points back or rewrite my essay?"
I need help creating a comprehensive Anki deck system for my 8-year-old who is following a classical education model based on the trivium (grammar stage). The child has already: - Mastered numerous Latin and Greek root words - Achieved mathematics proficiency equivalent to US 5th grade - Demonstrated strong memorization capabilities
Please create a detailed 12-month learning plan with structured Anki decks covering:
1. Core subject areas prioritized in classical education (specify 4-5 key subjects) 2. Recommended daily review time for each deck 3. Progression sequence showing how decks build upon each other 4. Integration strategy with existing knowledge of Latin/Greek roots 5. Sample cards for each deck type, including: - Basic cards (front/back) - Cloze deletions - Image-based cards (if applicable) - Any special card formats for mathematical concepts
For each deck, please provide: - Clear learning objectives - 3-5 example cards with complete front/back content - Estimated initial deck size - Suggested intervals for introducing new cards - Any prerequisites or dependencies on other decks
Additional notes: - Cards should align with the grammar stage focus on memorization and foundational knowledge - Please include memory techniques or mnemonics where appropriate - Consider both verbal and visual learning styles - Suggest ways to track progress and adjust difficulty as needed
Example of the level of detail needed for card examples:
Subject: Latin Declensions Card Type: Basic Front: 'First declension nominative singular ending' Back: '-a (Example: puella)'
https://gist.github.com/rahimnathwani/7ed6ceaeb6e716cedd2097...
> “Sum, es, est, sumus, ________, sunt”
That's not made for an 8-year old.
The prompt is:
Write an epic narration of a single combat between Ignatius J. Reilly and a pterodactyl, in the style of John Kennedy Toole.
I suppose creative writing isn't the primary selling point that would make users upgrade from $20 to $200 :)
Write me a review of "The Malazan Book of the Fallen" with the main argument being that it could be way shorterhttps://chatgpt.com/share/67522170-8fec-8005-b01c-2ff174356d...
It's a bit overwrought, but not too bad.
Interesting! I intentionally edit my keyboard layout to include the em-dash, as I enjoy using it out of sheer pomposity—I should undoubtedly delve into the extent to which my own comments have been used to train GPT models!
I use it all the time because it's the "correct" one to use, but it's often more "correct" to just rewrite the sentence in a way that doesn't call for one. :)
Did ChatGPT write this comment for you?
https://openreview.net/forum?id=FBkpCyujtS&nesting=2&sort=da...
(And two hyphens to en dash.)
Other things are the overuse of transition words (e.g., "however," "furthermore," "moreover," "in summary," "in conclusion,") as well as some other stuff.
It might not be fair to people who write like that naturally, but it is what it is in the current situation we find ourselves in.
https://www.reddit.com/r/ApplyingToCollege/comments/1h0vhlq/...
On Windows its Windows Key + . to get the emoji picker, its in the Symbols tab or find it in recents.
En dash is Alt+0150 and Em dash is Alt+0151
This is a huge improvement over previous GPT and Claude, which use the terrible "space, hyphen, space" construct. I always have to manually change them to em-dashes.
This shouldn’t really be a serious issue nowadays. On macOS it’s Option+Shift+'-', on Windows it’s Ctrl+Alt+Num- or (more cryptic) Alt+0151.
The Swiss army knife solution is to configure yourself a Compose key, and then it’s an easy mnemonic like for example Compose 3 - (and Compose 2 - for en dash).
For now, as ai power increase, ai powered ai writing detection tool also gets better.
Give it a few weeks for them to classify its outputs, and they won't have a problem.
I'm a professional writer and use em-dashes without a second thought. Like any other component of language, just don't _over_ use them.
Could be a tell for emails, though.
First impressions: The new o1-Pro model is an insanely good writer. Aside from favoring the long em-dash (—) which isn't on most keyboards, it has none of the quirks and tells of old GPT-4/4o/o1. It managed to totally fool every "AI writing detector" I ran it through.
It can handle unusually long prompts.
It appears to be very good at complex data analysis. I need to put it through its paces a bit more, though.