> The ballast in our application is a large allocation of memory that provides stability to the heap.
> As noted earlier, the GC will trigger every time the heap size doubles. The heap size is the total size of allocations on the heap. Therefore, if a ballast of 10 GiB is allocated, the next GC will only trigger when the heap size grows to 20 GiB. At that point, there will be roughly 10 GiB of ballast + 10 GiB of other allocations.
> For those interested, there is a proposal to add a target heap size flag to the GC which will hopefully make its way into the Go runtime soon.
What's wrong with the existing parameter?
I'm sure they aren't going this far to avoid all environment config without a good reason, but any good reason would be a flaw in some part of their stack.
Also, it is almost trivial to edit the Go sources (they are included in the distribution) and rebuild it, which usually takes just a minute. So Go is really suited for your own experiments - especially, as Go is implemented in Go.
Well, parts of it. You can't implement "make" or "new" in Go yourself, for example.
I do, I'm just objecting to "Go is implemented in Go".
The stable compiler is permitted to use unstable features in stable builds, but only for compiling the compiler. In essence, there are some Rust features that are supported by the compiler but only permitted to be used by the compiler. Unsurprisingly, various non-compiler users of Rust have decided that they want those features and begun setting the RUSTC_BOOTSTRAP envvar to build things other than the compiler, prompting consternation from the compiler team.
[0] https://github.com/rust-lang/cargo/issues/6627 [1] https://github.com/rust-lang/cargo/issues/7088
https://golang.org/src/runtime/map.go
https://golang.org/src/runtime/slice.go
I don't see why you couldn't do something similar in your own Go code. It just won't be as convenient to use as the compiler wouldn't fill in the type information (element size, suitable hash function, etc.) for you. You'd have to pass that yourself or provide type-specific wrappers invoking the unsafe base implementation. More or less like you would do in C, with some extra care to abide by the rules required for unsafe Go code.
Obviously they consider spending 50% more on hardware is a worthwhile compromise for the gains they get (e.g. reduction of developer hours and reduced risk of security flaws or avoiding other effects of invalid pointers).
If you do a lot of allocations, the GC overhead rises of course, but also would the effort of doing allocations/deallocations with a manual managing scheme. In the end it is a bit trade-off, what fits the problem at hand best. The nice thing about Rust is, that "manual" memory management doesn't come at the price of program correctness.
As a consequence, the heap pressure of a Go program is not necessarily significantly larger than that of an equivalent C or Rust program.
In contrary, in e.g. C I can wrap two 32-bit fields in a struct and freely pass then anywhere with zero heap allocations.
Also, record types are not going to fix the pointer chasing problem with arrays. This is promised by Valhalla, but I'be been hearing about it for 3 years or more now.
Ruby 1.8.x wants to say "Hello"
> We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references.
So maybe this is one of those things that just doesn't come up in most cases? Maybe most services also generate enough garbage that that 2-minute maximum doesn't really come into play?
But (virtually) nobody is writing games in Go, so it's entirely possible that it's an unusual case in the Go ecosystem. Being an unsupported usecase is a great reason to switch language.
I started up KSP just now, and it was at 5.57GB before I even got to the main menu. To be fair, I hadn't launched it recently, so it was installing its updates or whatever. Ok, I launched it again, and at the main menu it's sitting on 5.46GB. (This is on a Mac.) At Mission Control, I'm not even playing the game yet, and the process is using 6.3GB.
I think a better takeaway is that you can get away with GC even in games now, because it sucks and is inefficient but it's ... good enough. We're all conditioned to put up with inefficient software everywhere, so it doesn't even hurt that much anymore when it totally sucks.
Is this true? Go was built specifically for C++ developers, which, even when Go was first release, was a pretty unpopular language for writing web services (though maybe not at Google?). That a non-trivial number of Ruby/Python/Node developers switched was unexpected. (1)
https://commandcenter.blogspot.com/2012/06/less-is-exponenti...
> Our target community was ourselves, of course, and the broader systems-programming community inside Google. (1)
IDK if this is true for earlier versions, but as of today C# has pretty clear rules: 16MB in desktop or 64MB in server (which type is used can be set via config) will trigger a full GC [1]. Note that less than that may trigger a lower level GC, but those are usually not the ones that are noticed. I'm guessing at least some of that is because of memory locality as well as the small sizes.
On the other hand, in a lot of the Unity related C# posts I see on forums/etc, passing structs around is considered the 'performant' way to do things to minimize GC pressure.
[1] https://docs.microsoft.com/en-us/dotnet/standard/garbage-col... [2] https://blog.golang.org/ismmkeynote
[1] https://blogs.unity3d.com/2018/11/26/feature-preview-increme...
Often something like Redis is used as a shared cache that is invisible to the garbage collector, there is a natural key with a weak reference (by name) into a KV store. One could embed a KV store into an application that the GC can't scan into.
Doesn't Go have something like this available? It's an obvious thing for garbage-collected languages to have.
You could maybe hack around the GC performance without destroying the aims of LRU eviction by batching additions to your LRU data structure to reduce the number of pointers by a factor of N. It's also possible that a Go BTree indexed by timestamp, with embedded data, would provide acceptable LRU performance and would be much friendlier on the cache. But it might also not have acceptable performance. And Go's lack of generic datastructures makes this trickier to implement vs Rust's BtreeMap provided out of the box.
This is something important to know before choosing a GC-based language for a task like this. I don't think "generating more garbage" would help, the problem is the scan is slow.
If Discord was forced to do this in pure Go, there is a solution, which is basically to allocate a []byte or a set of []bytes, and then treat it as expanse of memory yourself, managing hashing, etc., basically, doing manual arena allocation yourself. GC would drop to basically zero in that case because the GC would only see the []byte slices, not all the contents as individual objects. You'll see this technique used in GC'd languages, including Java.
But it's tricky code. At that point you've shucked off all the conveniences and features of modern languages and in terms of memory safety within the context of the byte expanses, you're writing in assembler. (You can't escape those arrays, which is still nice, but hardly the only possible issue.)
Which is, of course, where Rust comes in. The tricky code you'd be writing in Go/Java/other GC'd language with tons of tricky bugs, you end up writing with compiler support and built-in static checking in Rust.
I would imagine the Discord team evaluated the option of just grabbing some byte arrays and going to town, but it's fairly scary code to write. There are just too many ways to even describe for such code to end up having a 0.00001% bug that will result in something like the entire data structure getting intermittently trashed every six days on average or something, virtually impossible to pick up in testing and possibly even escaping canary deploys.
Probably some other languages have libraries that could support this use case. I know Go doesn't ship with one and at first guess, I wouldn't expect to find one for Go, or one I would expect to stand up at this scale. Besides, honestly, at feature-set maturity limit for such a library, you just end up with "a non-GC'd inner platform" for your GC'd language, and may well be better off getting a real non-GC'd platform that isn't an inner platform [1]. I've learned to really hate inner platforms.
By contrast... I'd bet this is fairly "boring" Rust code, and way, way less scary to deploy.
To be clear: I wasn't suggesting that generating garbage would help anyone. Only that in a more typical case, where more garbage is being generated, the two minute interval itself might never surface as the problem because other things are getting in front of it.
At least with code hacking around the GC's behavior that code ends up being portable across the ecosystem.
There doesn't seem to really be a good option here either way. This amount of tuning-by-brute-force (either by knobs or by code re-writes) seems to just be the cost of using a GC.
Or do you think it adds to the discussion?
It is worth it to read and understand: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
If these issues were more common, there would be more configuration available.
[EDIT] to downvoters: I'm not saying it's not an issue worth addressing (and it may have already been since they were on 1.9), I was just answering the question of "why this might happen"
GOGC=off
More details here: https://golang.org/pkg/runtime/
I think with that, you could turn off GC after startup, then turn it back on at desired intervals (e.g. once an hour or after X cache misses).
It's definitely risky though. E.g. if there is a hiccup with the database backend, the client library might suddenly produce more garbage than normal, and all instances might OOM near the same time. When they all restart with cold caches, they might hammer the database again and cause the issue to repeat.
CloudFront, for this reason, allocates heterogeneous fleets in its PoPs which have diff RAM sizes and CPUs [0], and even different software versions [1].
> When they all restart with cold caches, they might hammer the database again and cause the issue to repeat.
Reminds me of the DynamoDB outage of 2015 that essentially took out us-east-1 [2]. Also, ELB had a similar outage due to unending backlog of work [3].
Someone must write a book on design patterns for distributed system outages or something?
[0] https://youtube.com/watch?v=pq6_Bd24Jsw&t=50m40s
[1] https://youtube.com/watch?v=n8qQGLJeUYAt=39m0s
This is definitely a familiar problem if you rely on caches for throughput (I think caches are most often introduced for latency, but eventually the service is rescaled to traffic and unintentionally needs the cache for throughput). You can e.g. pre-warm caches before accepting requests or load-shed. Load-shedding is really good and more general than pre-warming, so it's probably a great idea to deploy throughout the service anyway. You can also load-shed on the client, so servers don't even have to accept, shed, then close a bunch of connections.
The more general pattern to load-shedding is to make sure you handle a subset of the requests well instead of degrading all requests equally. E.g. processing incoming requests FIFO means that as queue sizes grow, all requests become slower. Using LIFO will allow some requests to be just as fast and the rest will timeout.
I've read the first SRE book but having worked on large-scale systems it is impossible to relate to the book or internalise the advice/process outlined in it unless you've been burned by scale.
I must note that there are two Google SRE books in-circulation, now: https://landing.google.com/sre/books/
But, per other comments, there isn't any direct malloc/free behavior. It just provides tools to help you enable the compiler to determine that GC is not needed for some.
Rust will probably be faster because it benefits from optimizations in LLVM that Go likely doesn't have.
Go arrays of structs are contiguous, not indirect pointers.
IIRC I have used GitLab and Bitbucket and self-hosted Gitea instances the same exact way, and I'm fairly sure there was an hg repo in one of those. Don't recall doing anything out of the ordinary compared to how I would use a github URL.
Ouch, Go never ceases to amaze. The Bitbucket case[0] is even more crazy, calling out to the Bitbucket API to figure out which VCS to use. It has a special case for private repositories, but seems to hard-code cloning over HTTPS.
If only we had some kind of universal way to identify resources, that told you how to access it...
[0]: https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8...
Wow, that's sad. I'm glad it works seamlessly, don't get me wrong, but I was assuming I could chalk it up to defacto standards between the various vendors here.
Sometimes it means an easy thing in most other languages is difficult or tiresome to do in Go. Sometimes it means hard-coded values/decisions you can't change (only tabs anyone?).
But overall this makes for a language that's very easy to learn, where code from project to project and team to team is very similar and quick to understand.
Like anything, it all depends on your needs. We've found it suits ours quite well, and migrating from a Ruby code base has been a breath of fresh air for the team. But we don't have the same performance requirements as Discord.
These are two things that make a lot of sense at Google if you read why they were done.
But unless you're working at Google, I struggle to guess why you would care about either of these things. The first requires sacrificing anything resembling a reasonable type system, and even with that sacrifice Go doesn't really deliver: are we really supposed to buy that "go generate" isn't a compilation step? The second is sort of nice, but not nice enough to be a factor in choosing a language.
The core language is currently small, but every language grows with time: even C with its slow-moving, change-averse standards body has grown over the years. Currently people are refreshed by the lack of horrible dependency trees in Go, but that's mostly because there aren't many libraries available for Go: that will also change with time (and you can just not import all of CPAN/PyPy/npm/etc. in any language, so Go isn't special anyway).
If you like Go for some aesthetic of "simplicity", then sure, I guess I can see how it has that. But if we're discussing pros and cons, aesthetics are pretty subjective and not really work talking about.
I like Go and I consider it a simple language because:
1. I can keep most of the language in my head and I don't hit productivity pauses where I have to look something up.
2. There is usually only one way to do things and I don't have to spend time deciding on the right way.
For me, these qualities make programming very enjoyable.
You mean where I explicitly said that "simple" didn't mean anything, so we should talk about what we mean more concretely?
> 1. I can keep most of the language in my head and I don't hit productivity pauses where I have to look something up.
The core language is currently small, but every language grows with time: even C with its slow-moving, change-averse standards body has grown over the years.
> 2. There is usually only one way to do things and I don't have to spend time deciding on the right way.
Go supports functional programming and object-oriented programming, so pretty much anything you want to do has at least two ways to do it--it sounds like you just aren't familiar with the various ways.
The problem with having more than one way to do things isn't usually choosing which to use, by the way: the problem is when people use one of the many ways differently within the same codebase and it doesn't play nicely with the way things are done in the codebase.
This isn't really a criticism of Go, however: I can't think of a language that actually delivers on there being one right way to do things (most don't even make that promise--Python makes the promise but certainly doesn't deliver on it).
I've been happy working with it for a year now, though I've had the chance to work with Kotlin and I have to say, it's very nice too, even if the parallelism isn't quite easy/ convenient to use.
1. map/filter are 2 lines of code each. 2. Inheritance is part of mainstream OOP, but there are some less common languages that don't support inheritance in the way you're probably thinking (i.e. older versions of JavaScript before they caved and introduced two forms of inheritance). 3. Generics are more of a strong type thing than an OOP thing.
Have you ever worked in a code base with many contributors that changed over the course of years? In my experience it always ends up a jumble where indentation is screwed up and no particular tab setting makes things right. I've worked on files where different lines in the same file might assume tab spacing of 2, 3, 4, or 8.
For example, say there is a function with a lot of parameters, so the argument list gets split across lines. The first line has, say, two tabs before the start of the function call. The continuation line ideally should be two tabs then a bunch of spaces to make the arguments line up with the arguments from the first line. But in practice people end up putting three or four tabs to make the 2nd line line up with the arguments of the first line. It looks great with whatever tab setting the person used at that moment, but then change tab spacing and it no longer is aligned.
some_function(arg1, arg2, arg3, arg4,
arg5, arg6);
some_function(
arg1,
arg2,
arg3,
arg4,
arg5,
arg6,
);
rustfmt looks to try and be "smarter" as it will move the argslist and add linebreaks to it go not go beyond whatever limit is configured on the playground, gofmt apparently doesn't insert breaks in arglists.
Instead, format arguments in a separate block:
some_function(
arg1, arg2, arg3, arg4,
arg5, arg6);
My own preference is tabs, because of less visual noise in code diff [review].
Not all editors, however, support this style of alignment, even if they support plugins (looks at vim and its forks).
Consider linting tools in your build.
The Azul guys get to claim that you don't need to tune their gc, golang doesn't.
Not sure how that makes the golang position any better.
See:
https://stackoverflow.com/questions/19094704/indentation-in-...
Thing is, few if any IDEs support the concept, so if I want to have half-indents, I must use spaces. Unfortunately, these days that means giving up and using a more common indent style most of the time, as the extra bit of readability generally isn't worth losing automatic formatting or multi-line indent changes.
BTW when you mix spaces with tabs you eliminate all benefits that tabs give (for example you no longer can dynamically change tab size without ruining formatting.
No, I used to be a notepad user (on personal projects, not shared work) (you can kinda see it in the use of indentation to help convey information that IDEs would use font or colour to emphasize), and these days use tabs but longingly wish Eclipse, etc. had more options in their already-massive formatting configuration dialogues.
<tab>(inserts 4 spaces)<tab>(replaces 4 spaces into a tab that is 8 columns)<tab>(adds 4 spaces after the tab)<tab>(replaces with two tabs and so on)
Unless I misunderstood what formatting you were using.
> We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change the garbage collector GC Percent on the fly. Unfortunately, no matter how we configured the GC percent nothing changed. How could that be? It turns out, it was because we were not allocating memory quickly enough for it to force garbage collection to happen more often.
As someone not too familiar with GC design, this seems like an absurd hack. That this 2-minute hardcoded limitation is not even configurable comes across as amateurish even. I have no experience with Go -- do people simply live with this and not talk about it?