I’m not saying that it’s a perfect solution, and yes, using C memory defeats the purpose of memory safety. But it’s a trade-off, as most engineering things are. Rewriting in Rust is also a gigantic trade-off along many many more axes, and Rust isn’t immune to people using `unsafe` in practice either. In both Go and Rust, it’s possible to carefully engineer so as to isolate where sources of memory unsafety are, either through the language or through principled policy and software design.

You make an excellent point i just wanted to correct your usage of "unsafe" in rust: It does not signify "im writing c" now as many invariants are still upheld in Rust even with the usage of unsafe. Id say its still much "safer" than using unmanaged memory in go as you have far more facilities you may use and more invariants are upheld.

Also, they didn't have to do this.

I really am glad Discord did not go with this route. It's a chatting application, which means you're dealing with a TON of user generated content. Unmanaged memory and user generated content is an absolutely terrible idea.

...?

The article is about a very specific service handling state for if you've seen a message in a channel. It has nothing to do with processing user content.

The problem with go here is (a) garbage collection, such that rust likely wouldn't have needed any unsafe outside of well-known libraries; and worse (b) when you need to do stupid C shit in go, there's no `unsafe` to contain it; it just spews over the rest of your code.

"Principled policy and software design" is exactly what's needed to get rid of memory bugs in C too! So we can conclude that these things are impossible, or else we would not need memory safe languages in the first place. Since it's increasingly apparent that go requires these mythical safeguards after all, it's safe to say that it is not a memory safe language in the useful sense.

In general, I think this all-or-nothing attitude paired with FUD language is detrimental to good engineering. Memory unsafety is generally bad, I get it. But it’s exaggerating to say that these things “spew” all over codebases, or that you’re entering some inescapable memory hell that has no way to be treated properly. The original proposal was to manage some memory yourself within Go to solve one very specific problem with a very specific module of code. That’s not a lot of surface area and has a clear API. It doesn’t seem unfathomable that qualified, highly paid engineers can’t do a good job solving that problem comprehensively without rewriting everything else.

I think the question is why not rewrite just the LRU cache in C/C++/Rust (whatever floats your boat), as opposed to rewriting everything in Rust.

One reason would be that doing that would require switching over to cgo, at which point you deal with the PITA that is an FFI, and the PITA that is cgo (it's easy to enable but it drastically alters your "game plan").

And here the cache is essentially the entirety of the micro-service:

> [The service's] sole purpose is to keep track of which channels and messages you have read.

so it's a giant map of (user, object) -> read flag or something along those lines. The DB commit and service input would be very light, and not really your main concern.

The question I’d have is “why use a custom cache at all”, this sounds like a problem that has a whole pile of battle-tested off-the-shelf solutions.

The original implementation of this "service" was in Redis using lua scripts.

We hit the performance ceiling of redis, and rather than deciding to go with redis-cluster, and dealing with the headache of managing persistent redis, we went this way, so we could consolidate data storage to scylladb/cassandra (it was cassandra at the time of rewrite, scylla now since we've migrated our data stores over).

any reason to not consider Aerospike for Cache?

We don't have any operational experience with Aerospike.

We have a bunch of operational experience with C*/ScyllaDB.

That sounds like basically what they did; they rewrote the ReadStates service in Rust and it sounds like that's basically a wrapper around an LRU cache. The rest of Discord is still Go/Elixer/whatever else they use.