Instrumentation: The First Four Things You Measure

Re:Instrumentation: The First Four Things You Measure by ackerman80 on Thursday January 26, 2017 @04:32PM (#13495418)

Came across this which gives good insight into the 4 golden signals for a top-level health tracking: https://blog.netsil.com/the-4-golden-signals-of-api-health-a...

One thing of note in the graph is the tracking of response size. This would be very useful for 200 responses with "Error" in the text. Because then the response size would drop drastically below a normal successful response payload size.

In addition to Latency, Error Rates, Throughput and Saturation , folks like Brendan Gregg @ Netflix have recommended tracking capacity.

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by maplebed on Thursday January 26, 2017 @04:35PM (#13495442)

Are you a plant? It must just be coincidence that the second post in the series is titled "measuring capacity." :) https://honeycomb.io/blog/2017/01/instrumentation-measuring-...

(bias alert - I work on Honeycomb)

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by ackerman80 on Thursday January 26, 2017 @05:50PM (#13496092)

we are all learning from the same folks ahead of us it seems :)

I agree with other comments though the devil is in the details of how to actually setup these "golden signals" so that they are useful and not just drown everyone in packet level non-sense.

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by saravana87 on Thursday January 26, 2017 @06:51PM (#13496522)

TCP retransmission rates looks like a useful metric which can help in monitoring the health of a service. One way to obtain that is by analyzing service interactions as mentioned in the blog. Tracing could be another way through which we can find that info. I am curious as to how code instrumented monitoring solutions get that information. (PS: I work for Netsil)

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by bbrazil on Thursday January 26, 2017 @08:01PM (#13496882)

By default you can only get that per-kernel from /proc/net/netsnmp. BPF may allow something more granular.

The other way of approaching it is to look for the additional latency it causes, which you can spot on a per-service basis.

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by saravana87 on Friday January 27, 2017 @01:53AM (#13498218)

Additional latency could be an indicator, but there's no guarantee that it is because of retransmissions ?

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by bbrazil on Friday January 27, 2017 @04:35AM (#13498724)

If you look at your latency histogram and are seeing a bump at around 200ms above normal (which was the default minimum wait time a few years back anyway), it's probably retransmits.

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by saravana87 on Friday January 27, 2017 @01:12PM (#13501436)

Got it.

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by rboyd on Thursday January 26, 2017 @07:53PM (#13496855)

you can get retransmits from 'sar' on linux

[ Parent | on HN ]

Re:Instrumentation: The First Four Things You Measure by saravana87 on Friday January 27, 2017 @01:46AM (#13498202)

I see. But, it looks like it is per host and there is no way to find out for a particular service running on the host.

[ Parent | on HN ]