News for hackers. Stuff that matters.
Re:Gemini 4 Neon by foohis on Wednesday November 18, 2026 @05:16PM (#50553034)
Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)

Other companies would use completely raw internet data for pre-training, which is basically trash. Anthropic's secret sauce is that they filtered and edited down their data for only the highest-quality material.

Meanwhile its obvious that companies like Google would use trash data like email and chat logs for pre-training.

This is a classic case of "less is more" and "garbage-in-equals-garbage-out"

Re:Gemini 4 Neon by quixoticelixer- on Wednesday November 18, 2026 @05:30PM (#50553231)
That doesn't really make sense because as good as Opus 5.5 is it writes awful terrible, disgusting slop.
Re:Gemini 4 Neon by foohis on Wednesday November 18, 2026 @05:39PM (#50553353)
It's a much, much better writer than literally every other model.
Re:Gemini 4 Neon by quixoticelixer- on Wednesday November 18, 2026 @07:35PM (#50554427)
It's better than Opus 5 but it still has that incredibly punchy contrasty LLM style that worse than most other models
Re:Gemini 4 Neon by thriving_earthw on Wednesday November 18, 2026 @09:39PM (#50555386)
I sometimes use Gemini for quick searches or stuff and the writing style is such a breath of fresh air. I mean I've been so locked in to the anthropic ecosystem because I find that code, the desktop app, and the ability of their models to understand what I mean and execute so well is incredible. But sometimes I'll read something it writes and I'll want to throw my computer at the wall. I just become apoplectic with rage for a second before I of course cool down and move on.
Re:Gemini 4 Neon by BigGreenJorts on Wednesday November 18, 2026 @10:00PM (#50555518)
I've been feeling this more and more lately as I've been trying to delegate technical writing to it. It's so horrifically repetitive, verbose and somehow non specific I can't even really use it as a base to edit down to something legible.
Re:Gemini 4 Neon by weitendorf on Wednesday November 18, 2026 @05:44PM (#50553424)
Anthropic doesn't want people to use their models for the chat/persona style use cases OpenAI has spent more time and effort optimizing for.

If you look back at GPT2 and the early days of AI safety/risk this was actually something the kinds of AI researchers that joined Anthropic didn't want - RL on coding->using that accelerate development and to bootstrap capabilities across domains was always the plan.

I personally find that Opus 5.5 and Fable 5+ are generally fine to chat with but just very wordy and culturally "LessWrong/essay-guy". Which, if a model is going to have a personality, is close to the one that I want (though it does piss me off when it gets preachy about standards Anthropic clearly doesn't hold themselves against)

I also want to immediately recognize AI generated content on the web or from another person, so I'm very glad it has a distinct voice

Re:Gemini 4 Neon by quixoticelixer- on Wednesday November 18, 2026 @06:17PM (#50553742)
Yeah maybe that's the case, and maybe I am more sensitive than other people. But I don't really use AI for that sort of thing but even for technical stuff I still find it's writing style a real turn off. Gemini still has lots of AIisms but it's a lot more tolerable.
Re:Gemini 4 Neon by LeoPanthera on Wednesday November 18, 2026 @06:12PM (#50553686)
If the training set is smaller, doesn't that make sense though? There will be much less variety in its style and so its going to make very recognizable output.
Re:Gemini 4 Neon by quixoticelixer- on Wednesday November 18, 2026 @06:16PM (#50553731)
Yeah that could be it. I guess the thing about what makes AI writing so repulsive sometimes is not that it's bad like a human would be bad, but that it does the same things over and over.
Re:Gemini 4 Neon by impulser_ on Wednesday November 18, 2026 @05:40PM (#50553369)
I don't think it pre-training, because surely these labs all have similar data and Google should be very good at ingesting, cleaning, and filtering data.

I honestly think it just their post-training is fantastic and I think it has something to do with them being very opinionated on how they want the model to act that it basically the best at roleplaying a human. Humans are very good at completely and figuring out things that there aren't "trained" on if given basic set of intelligence. So when it working on these long tasks that might not be directly in it training data it still reason through it very well.

They got made fun of for this in the pass because they do a lot of research on this including hiring a philosopher.

This is just my best guess, because you don't see any other labs talk about how they made the model act and Anthropic is basically obsessed with it and talks about it every model release.

I think they just have better "taste" and that taste is their secret sauce.

Re:Gemini 4 Neon by Topfi on Wednesday November 18, 2026 @06:01PM (#50553592)
Purely outside looking in, but yes, I'd argue you are right and higher-quality is a major part of it, both for pre- and post-training. There appears to be a far greater willingness to adopt specifically tailored training sets from third-parties, consistently update and build on them.

From simple frontend work to strategically adding virtual rulers when working with LIDAR scans to extract more accurate data, there are very clear points when Anthropic introduces a new external data set into their mix and while I have seen the same from competitors such as Google, there simply appears to be a greater tendency to stick with these once they have them.

As an example, at release it was quite apparent that Gemini 3 Pro, 3 Flash and Opus 4.5 had a similar, possible even the same frontend-focused training base bought in from a third party. But while Anthropic has had both smaller and larger improvements across each subsequent Opus release, I have not noticed such a consistent improvement with Gemini models over the last year. Maybe Anthropic simply buys updates more often, maybe they create more artificial PRs on top, who knows.

Same with the measurements, there was a point a few months ago when models started to exhibit a very systematic way of handling visual references in some few leather-crafting tasks, enabling whole new ways of working more efficiently combining existing measurements with images. Step-by-step pixel measurements, compensation for angles and poor captures, etc. Have since seen this in other models, again very much feels like a bought in data set that they are pushing as far as it goes.

Then there is 3D capability with models like Muse Wonder Full (have not used it myself, if it ain't on API, it doesn't exist), which I have seen PRs off I do find very parsable and useful, but which is terrible at any modelling task because these capabilities do not emerge from just training on vast amounts of data, they require high-quality, well tailored, step-by-step reference data of the entire process.

Re:Gemini 4 Neon by hackingonempty on Wednesday November 18, 2026 @06:20PM (#50553755)

  Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)
  [ ... ]
  Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.
Google has been scanning every book they can get their hands on for 20 years or more. They have a bunch of patents on non-destructive book scanning gear. Surely they are using that data to train their models.
Re:Gemini 4 Neon by wahnfrieden on Wednesday November 18, 2026 @06:57PM (#50554079)
Surely they are destroying scanned books for training
Re:Gemini 4 Neon by mastax on Wednesday November 18, 2026 @09:36PM (#50555364)
They invented those nondestructive scanners before the court cases that effectively mandated destruction.
Re:Gemini 4 Neon by wahnfrieden on Wednesday November 18, 2026 @11:18PM (#50556031)
Why does that allow them to use the scans for LLM training? Those scans were used for a different purpose. I don’t understand the logic.
Re:Gemini 4 Neon by foohis on Wednesday November 18, 2026 @07:21PM (#50554290)
won't help if they also include trash internet text in the training set as well.
Re:Gemini 4 Neon by etoxin on Wednesday November 18, 2026 @07:57PM (#50554636)
it's kinda known that the gemini models write better english which goes against the "trash data" remarks.
Re:Gemini 4 Neon by Gareth321 on Thursday November 19, 2026 @01:17AM (#50556653)
Maybe not. If they’re trained on real world emails and Reddit comments, they’ll write more like “real” people, but they’ll be dumber.
Re:Gemini 4 Neon by nl on Wednesday November 18, 2026 @06:24PM (#50553789)
This is incorrect.

Everyone uses books as well as internet data.

Anthropics difference is they have a huge data budget for buying RL environments for real world tasks.

Re:Gemini 4 Neon by positr0n on Wednesday November 18, 2026 @09:27PM (#50555307)
I get recruiter emails for $150/hr to do code review training for anthropic (W-2, although short duration contract).
Re:Gemini 4 Neon by nl on Wednesday November 18, 2026 @11:47PM (#50556179)
Yes, and if you look on LinkedIn there are many companies doing similar tasks in areas from accounting to physics to HR. They sell to all companies, but Anthropic seems the most aggressive. This from last year:

> According to The Information, leaders at Anthropic have discussed spending more than $1 billion on RL environments over the next year.

https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on...

Re:Gemini 4 Neon by jonah on Thursday November 19, 2026 @01:08AM (#50556601)
For other fields like medical or legal it's 2-3x that.
Re:Gemini 4 Neon by UltraSane on Wednesday November 18, 2026 @06:28PM (#50553822)
I think by now everyone training trillion parameter models is using almost every book, magazine, research paper, etc they can get.
Re:Gemini 4 Neon by mlmonkey on Wednesday November 18, 2026 @06:58PM (#50554092)
Microsoft (MSAI) showed that quality >> quantity with their Phi model a couple of years ago. It was a small model, but trained on very high quality data, and did better than much larger models.
Re:Gemini 4 Neon by amarcheschi on Wednesday November 18, 2026 @07:39PM (#50554469)
I can't say what precisely, but other big companies are also doing what they can to get human data of high quality, just look for all the companies that popped like mushrooms that do ai reinforcement tasks, micro1, ethos, mercor ecc
Re:Gemini 4 Neon by olalonde on Wednesday November 18, 2026 @08:13PM (#50554760)
Is this speculation or do you have a reliable source for that?
Re:Gemini 4 Neon by mrheosuper on Wednesday November 18, 2026 @09:27PM (#50555308)
Is that reason why Claude talks so like .... a nerd.
Re:Gemini 4 Neon by potamic on Wednesday November 18, 2026 @11:33PM (#50556116)
Is this your opinion or you have a source for your claim?
Re:Gemini 4 Neon by nikhilsimha on Thursday November 19, 2026 @12:24AM (#50556372)
i saw a researcher at uw claim that claude is trained on genome sequences and that others arent.

genome sequences are probably very dense in “reasoning”

Re:Gemini 4 Neon by oasisbob on Thursday November 19, 2026 @12:43AM (#50556465)
I have a weak theory that some of the strange contortions in Claude come from training on scientific literature.