More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill.
Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates.
Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons.
At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time.
Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.
Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following:
1. Delivering features faster
2. Delivering features with meaningfully higher quality
3. Delivering the same features with lower headcount costs
(2) is hard to measure, so you can generally only sell on (1) and (3). During the 80s the business world learned the hard way you can throw away a lot of money on useless efficiency. If that interests you I recommend "The Goal" by Goldratt for a fictionalized account of those learnings.
* There's also a subtler point that corporate finances may mean that even if efficiency is perfectly captured, a 1% efficiency increase may only be worth 0.1% of your salary.
I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk.
One security incident response will pay for the entire splunk infrastructure.
Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal.
So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors.
Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.