I had this thought yesterday, the era of developers picking tech is probably over. Microsoft's wheelhouse is selling to upstairs.
They're going to eat everybody's lunch that is still playing the old game of selling to the devs.
It's the modern era version of "Nobody ever got fired for buying IBM"
Don't really know when that era ever existed
"My budget gives me 100 million tokens for GTP 6.1 SOL. How many tokens for this new model would I get? Let me quickly divide by 0.0126 in my mind to figure it out."
x times more/less is the simplest form of comparison because bigger value is divisible by lesser value. I'm saying this because we first teach children division and multiplication and only then fractions.
> Decision 1.1 provides model routing integrated right into your applications so that the bigger, pricier models are only used when you really need them. On the AA Decision Index Microsoft is ahead of Jev, Google and OpenAI, while being faster and cheaper. Used by 4,000 customers, it’s 50x faster and 79x cheaper than GPT-6.1 Sol.
John here from the team that built Microsoft Decision 1 and 1.1.
We compare ourselves to other decision models in our blog post on Microsoft's Command Line blog from November 17 [1]. We name all of the public benchmarks we evaluated against in the post, and provide a breakdown by category.
We also are on the Artificial Analysis leaderboard, which is an independent third party assessment. On AA, we score 68, above Gemini Decision 1, GPT-6 Luna Decisions, and Jev 1.14. [2]
Our previous MD1 model from October is also benchmarked on JevBench, another independent benchmark for decision models. [3] On JevBench, we are 9th place for API models, ahead of Gemini Decision 1 and Jev 1.14. We look forward to seeing how the new MD1.1 scores on JevBench.
With regards to cost: Our models are the same price as Jev on a per-token basis. However, on both JevBench and Artificial Analysis benchmarks, our models use fewer tokens per task than Jev. For example, our cost score for MD1.1 on Artificial Analysis at $140 per 1M tasks compared to Jev's $156 and Gemini's $345. There are similar stats for latency, where our new MD1.1 model is 2x faster than our previous one.
[1] https://commandline.microsoft.com/microsoft-decision-model-g... [2] https://artificialanalysis.ai/evaluations/artificial-analysi... [3] https://benchmarkheaven.com/jev-models/api