And a lot of that is marketing, keeping unused capacity in reserve, etc. Microsoft can probably afford a very low price without losing money if they price the compute at-cost. Certainly less than $7/request. 70 teraflop per request, and an $8000 GPU, with $2k of server overhead, $10k in power and cooling costs, $20k total for 312 teraflops of compute, leased for 2 years gives you an at-cost hardware price of $0.00007/request: https://www.google.com/search?q=%2420000*70/(312*3600*24*365... So even assuming just a 1% utilization rate, you’re still talking less than 1 cent per request. Remember, inference runs at reduced precision. FP16 (or even just 8 bit) is WAY cheaper than fp64 bit. More than an order of magnitude cheaper. 9.7Tflops vs 312Tflops, factor of 32 improvement.

With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity.

https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...

I think your compute cost figure is both very old and probably assumes double precision.