Are there really such architectural differences between for example sol and astra that makes astra twice more in token price? Or maybe token cost covers training expenses? For me it's hard to believe that astra need twice the computing power that sol needs...
The problem is how they inject it. Their “knowledge” isn’t natural language; it’s templated Wikidata triples like "X is the capital of Y." That’s a super low-entropy, highly repetitive distribution. When you cram enough of that into a fixed token budget, you’re not really teaching the model more facts — you’re just destroying linguistic diversity and skewing the token statistics.
In real pretraining or domain adaptation scenarios, “knowledge” tends to appear in richer, more varied contexts. The practical takeaway isn’t "don’t add too much domain data," but rather "don’t overrepresent any single format or narrow syntactic pattern" The issue seems more about representation homogeneity than about factual density itself.
I noticed that there are several keywords that seen on blog or heard in YouTube video give you almost 100% certainty that it is AI generated. Words I mean are not super widely used in English and they are "intricate", "streamline", "meticulously". I've noticed it's common for LLMs to use these words also when using it every day. Is it kind of bias? Or I am biased as non native English speaker?