News for hackers. Stuff that matters.
Re:Go Memory Management by TheDong on Monday August 20, 2018 @12:55AM (#17797809)
Spoilers for why the simpler program is different: It doesn't do networking, which means it doesn't use any cgo. The networking stack by default links against libc, but if you don't use it, your go program doesn't have to link against libc.

You can always configure the CGO_ENABLED or -tags netgo flag in order to cause networking to not use libc either.

Re:Go Memory Management by otterley on Monday August 20, 2018 @09:56AM (#17799874)
Networking doesn’t require cgo. Only programs that expliticly call C functions (i.e. those that use ‘import “C”’) do.
Re:Go Memory Management by TheDong on Monday August 20, 2018 @11:11AM (#17800599)
It doesn't require cgo, but it defaults to it on linux.

To quote from the official docs: https://golang.org/pkg/net/#hdr-Name_Resolution

> When cgo is available, the cgo-based resolver is used instead under a variety of conditions: .... when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the Go resolver does not implement, and when the name being looked up ends in .local or is an mDNS name.

Almost any linux machine's `/etc/nsswitch.conf` will contain features that the go resolver doesn't implement. Because of that, almost any dns resolution will default to cgo.

In addition, selecting between netgo and cgo is a runtime feature by default (see GODEBUG=netdns=go|c), so it still has to build it in regardless unless it's explicitly disabled.

You can also determine this by using `ldd` on a go binary.

> Only programs that expliticly call C functions (i.e. those that use ‘import “C”’) do.

That is exactly what go uses in programs that use networking. Here's the 'import "C"' in question: https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417a...

Re:Go Memory Management by e12e on Monday August 20, 2018 @07:01PM (#17804901)
> Almost any linux machine's `/etc/nsswitch.conf` will contain features that the go resolver doesn't implement.

Do you know of a short list of top features missing? I would kinda hope that modern and generally useful features would be in netgo. And old rotted code, security risks and anachronisms would be abandoned. But from your tone, I gather that this is more than a "people only use 10%, but never the same 10%" type of thing?

Re:Go Memory Management by TheDong on Tuesday August 21, 2018 @12:28AM (#17806991)
Here's the code that determines if it needs to fall back to cgo: https://github.com/golang/go/blob/161874da2ab6d5372043a1f393...

Notably, it can only handle the following sources: files, dns, myhostname, mdns* (but only a subset of those if you read the gory details).

It doesn't handle the fairly uncommon "mymachines" or "resolve" (sometimes used in the systemd world nowadays)

I probably spoke too pessimistically there. Many machines won't have an nsswitch.conf that requires using cgo for common domains. After re-reading the code, it does handle more than I thought (mainly mdns).

> I would kinda hope that modern and generally useful features would be in netgo. And old rotted code, security risks and anachronisms would be abandoned

The fundamental problem here is not that. The nss system libc uses for network resolution is a pluggable system, and it chose dynamically linked c modules as its plugins.

It's not that they're old rotted insecure bits of code, it's that the interface is inherently a c-centric one, and go early on decided it wished to do syscalls directly instead of call them through libc (as literally every other programming language does).

Finally, that means go can't ever reach full parity as long as it wishes to avoid c, since anyone can simply write a new nss plugin in c and list it in resolv.conf, and go has no way of having that same arbitrary behaviour in go.

Re:Go Memory Management by e12e on Tuesday August 21, 2018 @02:47AM (#17807564)
Thank you for taking the time to reply.

> that means go can't ever reach full parity as long as it wishes to avoid c

Well, go has for better or worse decided on a "we'll bring our own batteries" approach - and that does imply a burden of re-inventing a lot of wheels. On the other hand, those wheels are all likely to be painted in just the right shade of gopher brown.

At a glance, nis/yp appear to be missing. Not sure if nis doesn't count as outdated though.

Re:Go Memory Management by TheDong on Monday August 20, 2018 @06:15PM (#17804511)
(note for anyone coming by in the future: that link is not the best one, https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417a... is where most of the cgo code for linux is)
Re:Go Memory Management by otterley on Monday August 20, 2018 @11:45AM (#17800875)
Thanks for the clarification!
Re:Go Memory Management by z92 on Monday August 20, 2018 @06:03AM (#17798740)
And I thought libc didn't use more than a few megs of ram. Here the difference seems to be like hundreds of megs.
Re:Go Memory Management by Arkanosis on Monday August 20, 2018 @08:36AM (#17799396)
It's virtual memory, not physical memory. One should really see virtual memory as “addresses available for use within a single process”, not “space to store actual data, shared by all processes on a computer”.

On modern 64 bits architectures, you have plenty of it to spend. To give an idea of how much of it you have: since Linux 2.6.11 on x86_64, one can address up to 128 TiB (that's 128 thousands of GiB) of virtual memory per process. Starting with Linux 4.12, one can address up to 128 PiB (that's 128 millions of GiB) of virtual memory per process. That's completely unrelated with how much physical memory you have.

Re:Go Memory Management by scott_s on Monday August 20, 2018 @09:34AM (#17799727)
Correct. When explaining it to students, I have used the metaphor of a housing developer building homes in a subdivision: going to the OS to "allocate" virtual memory is like getting a building permit from the government for a large swath of land. Having permission to build is not the same as a house actually being on that land (which is analogous to physical memory).

Much as housing developers are incentivized to get permits for a large block of land at a time, general purpose memory allocators are incentivized to ask the OS for permission to use large blocks of virtual memory at a time. Asking the OS for virtual memory is very slow, so to avoid doing it a lot, general purpose memory allocators ask for big chunks and manage those chunks themselves.

Re:Go Memory Management by JulianMorrison on Monday August 20, 2018 @07:24AM (#17799089)
Doesn't libc end up shared between all the things and amortized that way?
Re:Go Memory Management by de_Selby on Monday August 20, 2018 @08:03AM (#17799248)
Go programs are statically linked, you're referring to dynamic linking.
Re:Go Memory Management by Arkanosis on Monday August 20, 2018 @09:13AM (#17799617)
Actually, the libc is dynamically linked here, as shown in the /proc/maps section of the article.

So, to answer the GP: yes, resident memory from the libc ends up being amortized between all the processes that use it. But it doesn't even matter much as there's a mere 5 MiB of resident memory involved here. The “hundreds of MiB” are only of virtual memory, and that shouldn't be of any concern (see my other comment in this thread).

Re:Go Memory Management by zbentley on Monday August 20, 2018 @09:12AM (#17799608)
> Go programs are statically linked

Not necessarily true, unless you use strategies like the one suggested in the top level post above this. https://www.osso.nl/blog/golang-statically-linked/

In practice, those dependencies are ubiquitous enough that the "DLL hell" half of the dynamic-linking problem is avoided. The other issues (system-dependent behavior quirks, estimation from static libraries, linker caching, etc. etc.) remain in many cases.

Re:Go Memory Management by weberc2 on Monday August 20, 2018 @09:10AM (#17799591)
Go doesn’t statically link against libc; maybe it’s possible with the likes of musl, but in general, libc is dynamically linked.
Re:Go Memory Management by presscast on Monday August 20, 2018 @09:30AM (#17799707)
What's the trade-off when disabling CGO?