News for hackers. Stuff that matters.
Re:We Halved Go Monorepo CI Build Time by jiggawatts on Sunday June 26, 2022 @06:47PM (#31888843)
Recently I showed a development team how to use a debug tool that can automatically collect debug snapshots from production that can be opened with one click in the IDE. It’ll jump to the line of code and show the state of everything — not just the stack but the heap too.

My demo is to pick a random crash and diagnose the root cause while talking. The mean time to resolution is low single digit minutes.

I showed this to an entire team, one person at a time, solving bugs as I went. I showed the junior devs, senior devs, and their manager.

No interest. None. Just… silence.

The tools are amazing, but the lack of motivation from the typical developers for learning to use them is even more amazing.

Re:We Halved Go Monorepo CI Build Time by djbusby on Sunday June 26, 2022 @08:01PM (#31889239)
Pointer to the tool? It sounds awesome.
Re:We Halved Go Monorepo CI Build Time by jiggawatts on Monday June 27, 2022 @12:19AM (#31890545)
Azure App Insights can do Debug Snapshot collection, but it’s not the only such tool. It just makes the workflow quick and simple.

Ref: https://docs.microsoft.com/en-us/azure/azure-monitor/snapsho...

It works even better with DevOps source indexing added to the build pipeline:

https://docs.microsoft.com/en-us/azure/devops/pipelines/task...

A vaguely similar feature is Azure App Service memory leak diagnostic tools which take memory dumps on certain triggers or at intervals.

You can open these in Visual Studio and it’ll show the heap deltas over time.

Re:We Halved Go Monorepo CI Build Time by tomcam on Sunday June 26, 2022 @08:27PM (#31889378)
Seconded!
Re:We Halved Go Monorepo CI Build Time by laluser on Sunday June 26, 2022 @08:07PM (#31889268)
Playing devils advocate here. It could just be that they already have the stack trace in logs somewhere, so how does this help you any faster? Bulk of the time to resolution will be figuring out what to do anyways.
Re:We Halved Go Monorepo CI Build Time by jiggawatts on Sunday June 26, 2022 @08:30PM (#31889397)
These were all bugs that were nearly impossible to solve with just a stack trace. For example, there was a "format" exception on a web page with hundreds of uses of string formatting (to generate a report). Another example was a function call complaining about a null argument. Which instance? The page had 40+ calls to the same function, each with 6 arguments.

Most of these couldn't be reproduced either. As in, you'd get a crash once a day in a page that would otherwise work successfully thousands of times.

How would you fix a problem where there's a stack trace only from a release? The scenario is: you can't reproduce the errors, you don't get line numbers, you don't even get function argument values.

I could solve these in minutes using this tool. Could you match that without such tooling?

Re:We Halved Go Monorepo CI Build Time by randomifcpfan on Sunday June 26, 2022 @09:31PM (#31889720)
I’m guessing the organization incentives are against investigating/fixing random crashes. It might be unscheduled, or seen as “test” or “qa” or “ops” work. Try working with management to set up a way to reward the behavior you want to encourage.