News for hackers. Stuff that matters.
Re:Cloudflare Experiencing Latency Issues by jgrahamc on Thursday December 16, 2021 @04:35PM (#29584267)
I would have posted faster but I was too busy doing my own little bit of debugging which consisted of

    1. dig @1.1.1.1 jgc.org
    2. nc -v 104.22.11.223 80
    3. curl -v https://jgc.org/cdn-cgi/trace
    4. curl -v https://jgc.org/
Hmm #1 was fast so network is routing OK. Hmm #2 was fast so TCP is OK. Hmm #3 was fast so I know (because I worked on that code) that this code path is good. Hmm #4 is slow so that means component X is slow but still working.

Of course, in parallel I'm in a conference call with about 40 other people who have actual access to monitoring and systems and other things who can see exactly where things are.

But I was damn close with four commands and gave me confidence in what people were saying. But, I have to say, Cloudflare's internal distributed tracing system is pretty cool because I got sent a trace and you could see right where the slowdown was.

Re:Cloudflare Experiencing Latency Issues by tailspin2019 on Thursday December 16, 2021 @04:37PM (#29584286)
Now you're just showing off :)
Re:Cloudflare Experiencing Latency Issues by jgrahamc on Thursday December 16, 2021 @04:55PM (#29584506)
Allow an old man the fantasy that he still knows how the whole of the system works.
Re:Cloudflare Experiencing Latency Issues by maxgashkov on Saturday December 18, 2021 @04:50AM (#29603247)
That's a huge problem for companies of almost any scale. Can you shed some light on tools used internally in Cloudflare for tracing?
Re:Cloudflare Experiencing Latency Issues by jatone on Thursday December 16, 2021 @07:46PM (#29586194)
sounds like you do. ;)
Re:Cloudflare Experiencing Latency Issues by vinay_ys on Friday December 17, 2021 @12:58AM (#29588411)
What was component X? Was it a buggy rollout?
Re:Cloudflare Experiencing Latency Issues by jgrahamc on Friday December 17, 2021 @02:47AM (#29589132)
A proxy that serves traffic. No, it was way more complicated than “buggy rollout”.
Re:Cloudflare Experiencing Latency Issues by tuananh on Thursday December 16, 2021 @08:21PM (#29586504)
you often do you code these days?
Re:Cloudflare Experiencing Latency Issues by jgrahamc on Friday December 17, 2021 @02:47AM (#29589126)
Hardly ever. If I start a project I don’t end up having time to finish or maintain it which isn’t fair to the team.

If I write something it’s for my own use. And I like to write things that test Cloudflare. Доверяй, но проверяй.

Re:Cloudflare Experiencing Latency Issues by maxgashkov on Saturday December 18, 2021 @04:48AM (#29603236)
I see comrades you have on your team have taught you well /jk
[dead]
Re:Cloudflare Experiencing Latency Issues by dang on Thursday December 16, 2021 @08:23PM (#29586529)
First, please don't be an asshole and/or be snarky on HN. We don't want that here. (See https://news.ycombinator.com/newsguidelines.html.)

Second, "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." (also in https://news.ycombinator.com/newsguidelines.html)

Third, punishing users for discussing something work-related is one of the worst things you can do to damage this place. We want knowledgeable comments, and for most people, our work is what we know the most about. Disincentivizing that is a losing proposition (https://hn.algolia.com/?query=disincent%20by:dang&dateRange=...) so please don't.

Re:Cloudflare Experiencing Latency Issues by stillicidious on Friday December 17, 2021 @09:39AM (#29592749)
I'd suggest you are doing far more damage to the reputation of the board by conflating commentary with free marketing. The comment I replied to was the latter. If advertising is permitted where critique from actual paying users is openly punished, what is that saying about the board?

As for the extremely disappointing asshole comment, coming from a moderator is quite frankly astounding. "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith".

Re:Cloudflare Experiencing Latency Issues by jgrahamc on Thursday December 16, 2021 @04:41PM (#29584343)
We have whole teams of people to get status up and inform people and support them.
Re:Cloudflare Experiencing Latency Issues by stillicidious on Thursday December 16, 2021 @04:42PM (#29584353)
And yet I'm learning about this outage from HN. How did that happen?
Re:Cloudflare Experiencing Latency Issues by jgrahamc on Thursday December 16, 2021 @04:44PM (#29584382)
Suggest subscribing on https://www.cloudflarestatus.com/ as that was updated long before I came to HN to say anything (which was only after the incident was resolved).
Re:Cloudflare Experiencing Latency Issues by scrollaway on Thursday December 16, 2021 @04:50PM (#29584461)
Maybe it happened because you're busy commenting on HN instead of checking their status page.
Re:Cloudflare Experiencing Latency Issues by tailspin2019 on Thursday December 16, 2021 @04:46PM (#29584404)
Aviate Navigate Communicate.

They did all three of those in the space of a few minutes (if you were checking their status page - which you can subscribe to by the way).

Re:Cloudflare Experiencing Latency Issues by ithkuil on Thursday December 16, 2021 @06:00PM (#29585205)
That rule is great for one single person (a pilot). In corporate incident management we can do better, and not blocking communication because you're overwhelmed with keeping the system up; in fact proper incident management workflows designate different people with different roles and delegates communication duties to a dedicated person if possible.
Re:Cloudflare Experiencing Latency Issues by tailspin2019 on Thursday December 16, 2021 @06:13PM (#29585325)
> That rule is great for one single person (a pilot). In corporate incident management we can do better, and not blocking communication because you're overwhelmed with keeping the system up; in fact proper incident management workflows designate different people with different roles and delegates communication duties to a dedicated person if possible.

I agree with you, but Cloudflare seem to have done exactly what you're describing in this instance?

Re:Cloudflare Experiencing Latency Issues by ithkuil on Friday December 17, 2021 @02:16AM (#29588943)
Yes, my comment was about the Aviate Navigate Communicate rule, not about cloudflare's response