Re:Jev: New frontier model 40-400x cheaper and 20-200x faster
by ta988 on Tuesday September 15, 2026 @05:12PM
(#49718975)
Current models go beyond the simple embedding because you start to encode groups of concepts in the context-aware part of the model (attention heads or any other method). So it is never simply words/tokens in isolation anymore.
But in things like programming, most concepts are abstract - 'if hungry eat an apple' in programming terms would look like
'if hunger > 50 {apples--; hunger-=30;}'
and compilers work with 'concept erasure' - to them, tokens (which are like llm tokens) look like
'if var1 > 50 {var2--;var1-=30}'.
They don't care about how these things map to real concepts. So all the embedding directions used to encode real-world concepts are just noise to LLMs when programming. This greatly reduces dimensionality and training costs. So does a token representation tuned for programming constructs, rather than natural language would probably have a more efficient encoding.