In the article they describe their attempts at tuning and weren't happy with the results

The main issue with Go in this regard is that it generally doesn't have multiple ways of doing the same thing at least idiomatically. I mean sure, there are architectural choices you could make in a rewrite but the actual structure of the code itself is going to be very similar.

It does look like with some GC tuning (e.g. manually triggering GC's at a smaller interval than the Go automatic GC threshold) they might've mitigated the spikes, although I don't think they would have gotten the level of perf improvement they did. Golang assembly code IME is not very optimized compared to Rust/C++.

edit: reading comprehension skills are lacking, please see comment below for why I'm wrong

I don't understand.... isn't this idea (triggering GC more often) explicitly discussed in the article?

My understanding is they tried to tune the GC percent to make the automatic heuristic do GC sooner, but they didn't allocate enough to have that make a difference. However, Go has a way to manually trigger a GC which they could've set on a timer on a goroutine. If they weren't actually generating that much garbage then pause times should theoretically be pretty short if you're doing a GC every 5 seconds or something like that.

That being said it's not something that 100% is guaranteed to fix the issue so maybe they did test this and just didn't mention it in the blog.

Okay, I see what you're saying about the timer. That isn't in TFA, agreed.

But I still don't understand, because....

(NB: I'm not a GC expert, just a curious amateur, so my apologies if there are errors in the following, and the opportunity to be corrected in these errors is part of why I'm posting this.)

Regarding the "not much garbage => theoretically times would be shorter", my understanding is that this is actually not how GC works. The GC time is a function of the size of the GC pool, because GC works by walking ("tracing") the tree of live references. So the only way to make GC faster is to have not less garbage, but less stuff allocated at all.

Multi-generational GC works by dividing the whole pool into smaller pools, so that most GC passes only visit the high-churn nursery, but even then some GC passes need to read the

TFA mentions this, where they say "the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire [thing we were keeping track of]".

That is, they had virtually no garbage to collect, and that wasn't speeding up the GC. Which is consistent with how all tracing GC works, as far as I know.

Comments/corrections/clarifications are requested!!

I remember when this article came out, everybody was pointing out the fact that they used a go version that was several releases older.

Perhaps if the intent wasn't to convince their managers to let them write it in Rust, they would have tried using the latest Go version at the time?

Then why publish it at all?

Not to mention, the article made no effort to establish that it's describing the world 2 years prior to this being written

They actually would have mitigated it by upgrading their Go version, because the latest release at the time had a fix in the runtime that would have basically solved this.

Turns out no one on the team actually looked into issues in the Go repo to see if it was being addressed. Looks like they just wanted to write Rust, which is fine Rust is cool, but let’s not deceive ourselves.

That is not what happened, they did not publish the blog post immediately after the transition, and those changes to the GC did not happen until after the port happened. Some people made assumptions about timeline that were incorrect, and then repeated.

The discussion at the time on Reddit [1] mentions this. The general discussion as well talked about if the improvements, which were big in many cases, would have even improved this particular case. We’ll never truly know.

That said it is important to recognize that Go’s GC has received significant upgrades over the years, and remember that what’s true in the past may not be true today.

1: https://www.reddit.com/r/programming/comments/eyuebc/why_dis...

The point is that they could have found the issue and seen that it was about to be released. That would be good engineering, bad engineering is when you don't find the root cause of your problem and see if its being worked on

Good engineering is when you solve the problems you have. Sometimes there are multiple ways to solve a problem. Just because they did not choose the solution (which again, we're only speculating would actually solve the issue here, we don't have proof of that) you prefer does not make it poor engineering.