Re:It's tough being an Azure fan
by jiggawatts on Friday September 24, 2021 @12:25AM
(#28638618)
Log Analytics uses a columnar compression format on-disk, so ingested data is likely compressed by anywhere between 10:1 and 100:1, maybe even higher.
However, the wire format is super verbose JSON.
They bill per GB of the latter, not the former.
To put things in perspective: How many $ of CPU time do you imagine it takes to column-compress 1 TB of data? I would estimate that a single modern CPU core could do this in a minute or so. Factor in various inefficiencies and make it a super generous 1 hour. At spot pricing, that's about $0.01! One cent!!!
The larger cost would be bandwidth. Azure charges a huge markup for traffic (just like AWS), so for example zone-to-zone data costs $10 per terabyte at retail pricing (not internal costing).
They store that data for 30 days "for free" (lol). Assume a worst-case compression ratio of 10:1 and then that means that they have to retain 100 GB for 30 days. That's $9.43 for a Premium SSD at retail pricing.
So their hosting costs for Log Analytics is something like $20 per TB ingested, but they charge well over $2000 for it.
That 100:1 markup is pretty sweet if your KPIs are based on recurring revenue.
There is no way in hell they will ever "optimise" this. Any accidental improvement will be rolled back or "adjusted" to ensure the revenue stream doesn't fall off a cliff.
Have you not wondered why it's taken them so long -- over ten years -- to enable any feature to filter logs at the source?
However, the wire format is super verbose JSON.
They bill per GB of the latter, not the former.
To put things in perspective: How many $ of CPU time do you imagine it takes to column-compress 1 TB of data? I would estimate that a single modern CPU core could do this in a minute or so. Factor in various inefficiencies and make it a super generous 1 hour. At spot pricing, that's about $0.01! One cent!!!
The larger cost would be bandwidth. Azure charges a huge markup for traffic (just like AWS), so for example zone-to-zone data costs $10 per terabyte at retail pricing (not internal costing).
They store that data for 30 days "for free" (lol). Assume a worst-case compression ratio of 10:1 and then that means that they have to retain 100 GB for 30 days. That's $9.43 for a Premium SSD at retail pricing.
So their hosting costs for Log Analytics is something like $20 per TB ingested, but they charge well over $2000 for it.
That 100:1 markup is pretty sweet if your KPIs are based on recurring revenue.
There is no way in hell they will ever "optimise" this. Any accidental improvement will be rolled back or "adjusted" to ensure the revenue stream doesn't fall off a cliff.
Have you not wondered why it's taken them so long -- over ten years -- to enable any feature to filter logs at the source?