Re:Saving 13M Computational Minutes per Day with Flame Graphs
by tracker1 on Tuesday April 12, 2016 @03:07AM
(#11477724)
Profiling in general is a really good thing when you're seeing odd load/timing/performance issues... I once found a project was storing its' configuration settings (loaded/cached from DB) in a really badly performing way, an in-memory datatable, with text queries instead of a hashtable (not my design).
A single call wasn't so bad, but the lookup was happening many hundreds of times per request adding seconds to some requests. Wild how much difference a relatively small thing can make.
Re:Saving 13M Computational Minutes per Day with Flame Graphs
by surrealvortex on Wednesday April 13, 2016 @03:16AM
(#11486360)
That brings up another distinction - profilers don't distinguish between a method that takes very little time to run but is called very often and another method that is pretty expensive, but is not called very often.
Ultimately, we do care about the total time taken, but the approaches necessary for the two cases above are very different. In many cases, the method that is simply called very often will call for some type of caching solution in the caller, while the more expensive method will require retooling within the method itself.
Some 15 minutes of work improved CPU usage of my team's biggest fleet by ~40%. Considering we scaled up to 1500 c3.4xlarge hosts at peak in NA alone on that fleet, those 15 minutes kinda made my month :)
One thing to note once you eliminate the easy pickings is that as you go higher up the call graph, the profiler visualization is often misleading. There may be sections of code without safe-points, and stuff that appears wide on the flame graph may just be getting blamed for adjacent code that doesn't have safe points.