News for hackers. Stuff that matters.
Re:Why are my Go executable files so large? by PeterisP on Sunday December 29, 2019 @01:46PM (#21907609)
Is it just me or something like that runtime.pclntab shouldn't be included in production builds at all?

I mean, it makes all sense while you're developing and testing, but it should be reasonably possible to strip it out from production build binaries, and instead put it in a separate file so that if you do get a crash with a stack trace, then some external script can transform the program counters to line numbers, not have it embedded in every deployed binary.

Re:Why are my Go executable files so large? by benesch on Sunday December 29, 2019 @02:18PM (#21907830)
The Go language literally requires that pclntab be included in release builds. I'm with you—it seems kind of crazy that this was designed into the language—but there you have it.

The reason is that Go's standard library provides functions that allow retrieving a backtrace and symbolicating that backtrace at runtime:

* https://golang.org/pkg/runtime/?m=all#Callers

* https://golang.org/pkg/runtime/?m=all#CallersFrames

Unlike in C or C++, where you can link in something like libbacktrace [0] to get a best-effort backtrace, those Go functions are guaranteed to be correct, even when functions have been inlined. This is no small feat, and indeed programs compiled with gccgo will often be incorrect because libbacktrace doesn't always get things right when functions have been inlined.

[0]: https://github.com/ianlancetaylor/libbacktrace

Re:Why are my Go executable files so large? by PeterisP on Sunday December 29, 2019 @03:01PM (#21908072)
Is it that common for these functions to be used? Perhaps transitively through some popular libraries? Just being in the standard library doesn't even necessarily mean that these functions (and the data they need) should get included by the linker.
Re:Why are my Go executable files so large? by benesch on Sunday December 29, 2019 @04:44PM (#21908732)
Any program that uses a logging framework, including the stdlib log package, will wind up depending on runtime.Callers at least transitively. That’s probably most Go programs; certainly most of the programs large enough to be worrying about binary size.

Unlike in C, there are no macros like __FILE__ and __LINE__, so there is no alternative to runtime.Callers (short of preprocessing your Go source code).

Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @05:14PM (#21908939)
You can still get a backtrace without symbols though.

Why couldn't the Go team introduce a flag that strips symbols, while making clear to people that they should only use it if they are okay with backtraces looking like

  #1  0x00007ffff7ddb899 in __GI_abort () at abort.c:79
  #2  0x0000555555555156 in ?? ()
  #3  0x0000555555555168 in ?? ()
  #4  0x000055555555517d in ?? ()
  #5  0x0000555555555192 in ?? ()
or similar
Re:Why are my Go executable files so large? by giovannibajo1 on Sunday December 29, 2019 @06:05PM (#21909266)
Because one of the mantra of Go is not telling users to have a "debug" build and a "release" build. The development build is the one that goes into production, with no difference in optimizations, symbols and whatnot. This has pros and cons, like all tradeoffs.
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @06:23PM (#21909363)
Thanks; I didn’t know that, but it makes sense. Not a design decision I agree with, but it’s coherent, at least.
Re:Why are my Go executable files so large? by shaklee3 on Sunday December 29, 2019 @09:30PM (#21910302)
It seems like this can only be a con if other languages give you a choice. In other words, in C++ you can decide that your debug and release are the same, or you can make them different if you know what you're doing (like a good IDE does).
Re:Why are my Go executable files so large? by firethief on Sunday December 29, 2019 @10:35PM (#21910618)
Having more options is inherently better in C++, a language designed to reward mastery. Having fewer choices is better in Go, a language designed to make onboarding a new developer easy.
Re:Why are my Go executable files so large? by erik_seaberg on Monday December 30, 2019 @03:33AM (#21911732)
For any given tool you're going to spend most of your career at the top of its learning curve, so it doesn't make sense to weaken the tool to optimize for the brief period when you weren't.
Re:Why are my Go executable files so large? by firethief on Tuesday December 31, 2019 @01:36PM (#21923249)
As a developer I agree with you, but I think if I were a FAANG I'd be thinking: developers are crazy expensive, how can I commodify them? Every C++ developer is unique. There's endless variety of language subsets and build situations. It takes enormous investment to bring someone up to speed on a project, and only mandating a heavily-restricted subset of the language can make teaching it to fresh college grads practical. These are exactly the things Go is designed to avoid.
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @10:37PM (#21910631)
In my experience, new developers are not normally charged with determining the details of build flags, deployment systems, etc.
Re:Why are my Go executable files so large? by dodobirdlord on Sunday December 29, 2019 @11:10PM (#21910744)
Not to be too judgmental, because I recognize that every organization has its own way, but that sounds like a sign of organizational disfunction.
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @11:20PM (#21910780)
I just mean, when you join an existing project, most of that would already be set up.

In every project I’ve been on, I was asked to ramp up in the first few weeks by fixing small bugs here and there in the core codebase, not by setting up CI/makefiles/build infra/etc...

I actually think I might be totally missing your point, since I don’t really understand what you mean or how you arrived at the conclusion that it’s a sign of dysfunction.

Re:Why are my Go executable files so large? by HeWhoLurksLate on Monday December 30, 2019 @07:55AM (#21913017)
I hope that's how it would work for me- getting the task of setting up makefiles on the first day- before I know anything about the codebase- would feel pretty daunting, to say the least.
Re:Why are my Go executable files so large? by anthk on Monday December 30, 2019 @03:21AM (#21911673)
That's bullshit. C is simple and you have masterpieces written on it. And 6502 ASM is even more simple and yet some C64 games are a pure wizardry.
Re:Why are my Go executable files so large? by jasonhansel on Sunday December 29, 2019 @09:49PM (#21910386)
The "26M pclntab" issue sounds like an excellent argument against that design decision.
Re:Why are my Go executable files so large? by loeg on Sunday December 29, 2019 @10:01PM (#21910442)
Counterpoint: our work infrastructure for symbolizing stripped binary stacks sucks. I'd love to get backtraces with symbols from the field.

But also this entire section could be compressed and seekable and only expanded on demand. No need to leave it as a bloated uncompressed blob.

Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @10:11PM (#21910501)
> our work infrastructure for symbolizing stripped binary stacks sucks. I'd love to get backtraces with symbols from the field.

If stripped binaries don't work for you, nobody is saying you should be forced to use them. You would presumably just not avail yourself of the option.

(As an aside -- what's wrong with your symbolification infrastructure at work? This is something I spent significant time working on in a past job, so I'm always curious how other people are doing it).

Re:Why are my Go executable files so large? by loeg on Monday December 30, 2019 @08:19PM (#21918218)
My employer's internal infrastructure for this just sucks; it's not endemic to stripping binaries.

Specifics? The automation to associate cores with specific builds is bad or non-existent; the automation to load up GDB with the right files and corresponding sources or source branch is bad/non-existent; the fileserver storing symbol data is slow and sometimes remote to the developer across a very thin pipe; etc, etc. All to some extent foot-shooting by my organization. But we have TBs and TBs of build artifacts and hundreds of unique daemons producing cores and also have to debug kernel cores and multiple product branches.

Part of the pain is perhaps that we're cross-building for an embedded FreeBSD-derived system, but the majority of our developers use Linux or Mac, so we can't necessarily use host-native (or host-native-only) tools.

I think basically the situation is begging for someone familiar with the problem-space to sit down and bang out a FUSE filesystem or two (e.g. fetching sources on-demand as GDB reads them, instead of checking out a full multi-GB repo copy) along with a shell or Python script to load up a core. But no one has done it yet and management doesn't really prioritize developer tools.

Re:Why are my Go executable files so large? by ori_b on Sunday December 29, 2019 @11:22PM (#21910784)
Why? What problems is it causing you in practice?
Re:Why are my Go executable files so large? by roberson87 on Sunday December 29, 2019 @07:06PM (#21909599)
Are you sure this is true? Doesn't delve for example build Go source with special flags (gcflags=all='-N -l')to generate debugging symbols. I also remember having to build Go code with those flags for Stackdriver to get the correct debugging information without any optimisations.
Re:Why are my Go executable files so large? by sagichmal on Sunday December 29, 2019 @07:57PM (#21909868)
This shouldn't be necessary anymore.
Re:Why are my Go executable files so large? by roberson87 on Sunday December 29, 2019 @11:12PM (#21910752)
Thanks for the reply. Would appreciate if you could expand on this a little, since when? FWITW Google Stackdriver documentation [1] still states that you have to build source with those flags for version 1.1>.

[1]https://cloud.google.com/debugger/docs/setup/go

Re:Why are my Go executable files so large? by zer00eyz on Sunday December 29, 2019 @03:43PM (#21908322)
Backtrace can be amazing at run time if your going to put that info into logs. It makes finding "fringe" errors a lot less painful or more easily reproducible.
Re:Why are my Go executable files so large? by Too on Monday December 30, 2019 @03:22AM (#21911681)
Java and .NET can also get backtrace at runtime. Can't remember if they are symbolized or not but i think they are. How do they handle this?
Re:Why are my Go executable files so large? by thu2111 on Monday December 30, 2019 @03:54AM (#21911869)
They generate unwind information on the fly as they compile the code which is then stashed to one side, compressed and accessed only if a true language-level stack trace is required. Also they don't expose argument or return values in stack traces.
Re:Why are my Go executable files so large? by ComputerGuru on Monday December 30, 2019 @03:42AM (#21911787)
They’re interpreted/JIT’d and not compiled like golang.
Re:Why are my Go executable files so large? by pjmlp on Monday December 30, 2019 @03:51AM (#21911836)
They, have release and debug build modes.

.NET has AOT support since ever via NGEN, followed by Mono AOT, Xamarin,IL2CPP, Bartok, .NET Native and a couple of less know projects.

On Java side AOT has been always supported by JVMs for embedded deployments like Aonix (now part of PTC, PTC, Aicas, ExcelsiorJET, IBM J9.

And nowadays there is SubstrateVM as well, renamed recently as Graal Native Image, part of OpenJDK since Java 11.

Re:Why are my Go executable files so large? by andor on Monday December 30, 2019 @08:02AM (#21913063)
Not entirely true with regards to the question.

Yes, Java is interpreted and JITed, but on the bytecode level. When compiling sources to bytecode, line number information is written to class files.

Re:Why are my Go executable files so large? by loeg on Sunday December 29, 2019 @09:59PM (#21910434)
libunwind[1][2] should unwind correctly if a compiler emits correct DWARF CFI unwind table (required by C++ w/ exceptions; available for C in gcc/clang with -funwind-tables).

[1]: https://github.com/llvm-mirror/libunwind

[2]: https://www.nongnu.org/libunwind/

Re:Why are my Go executable files so large? by saagarjha on Monday December 30, 2019 @02:42AM (#21911537)
I don’t think libunwind gives you function names if you strip symbols and don’t have debugging information on hand.
Re:Why are my Go executable files so large? by loeg on Monday December 30, 2019 @08:23PM (#21918230)
Sure. libunwind is taking the place of pclntab here, not the symbol table. Go must also not strip symbols in order to print backtraces; if you want the same thing in C, don't strip the symbol table there either.
Re:Why are my Go executable files so large? by mynegation on Sunday December 29, 2019 @02:05PM (#21907733)
Not sure if it is just you but normally you DO want this information in the production build. It is quite a bad situation to have a run time exception in PROD and having no idea how it happened. Sure, there is defensive programming and checks and asserts but most of the time you cannot foresee everything.

I get the point about external symbols and location database, but oftentimes time is precious and having fully laid out stack trace in the log will allow you to get to the root much faster.

Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @02:28PM (#21907883)
> I get the point about external symbols and location database, but oftentimes time is precious and having fully laid out stack trace in the log will allow you to get to the root much faster.

You can also set up a service that automatically symbolicates everything in a log file as soon as it is generated, before a human ever even looks at it.

Granted, yes, this is slightly more complicated, but the point is that the toolchain should let the developers choose which strategy they want to use.

Re:Why are my Go executable files so large? by koffiezet on Monday December 30, 2019 @06:26AM (#21912605)
> let the developers choose

Absolutely not. Developers making choices about stuff like that is asking for troubles. I'm an SRE - and this is how production issues start, because some dev decided that somehow, in production, something like this should not be there, until problems pop up, but then it's too late.

For us it's simple, the moment it hits the CI/CD pipeline, it's a production build, even if it happens just end up on some test or staging environment. If you possibly need it there, you need it on production. Our way of working means that the exact same build artifact should be promotable to production.

Re:Why are my Go executable files so large? by crehn on Sunday December 29, 2019 @02:51PM (#21908022)
There is a big difference between choose and do.
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @05:15PM (#21908950)
I don't understand what you mean; can you elaborate?
Re:Why are my Go executable files so large? by mynegation on Sunday December 29, 2019 @06:37PM (#21909434)
Not a parent commenter but I interpreted that as: given choice between alternatives A (fast and good enough) and B (more complex and much better) you would want to choose B but end up doing A for various reasons (lack of time, unclear ROI etc)
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @06:43PM (#21909461)
Sure, but the toolchain designers can (relatively easily) offer both, so that people who really do have the resources and/or need to do B can do so.
Re:Why are my Go executable files so large? by firethief on Sunday December 29, 2019 @10:39PM (#21910639)
Are we still talking about a language that protects its users from the ability to write generics?
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @11:06PM (#21910729)
That’s a language feature; we’re talking about build flags. IMO, adding a build flag wouldn’t meaningfully change how simple the language is to learn and use, whereas there is at least a credible argument that generics would.
Re:Why are my Go executable files so large? by PeterisP on Sunday December 29, 2019 @02:56PM (#21908042)
There are very different production scenarios - in many of them noone will ever look (or even be able to look) at a stack trace if it crashes after it's shipped (at best you'll record bug reports from customers and attempt to reproduce them on your test hardware), so the debug information is literally useless there. And these are the same scenarios extra 50mb of disk and memory matter more than for some software running in a cloud environment.
Re:Why are my Go executable files so large? by erik_seaberg on Sunday December 29, 2019 @04:38PM (#21908689)
I was pleasantly surprised how good Microsoft's tooling was around firing up a debugger to examine the final state in a crash dump using external symbols from that build. Everything seemed to work except you couldn't resume any thread. I agree symbols don't need to be embedded in every running binary, but having a warm copy somewhere can be pretty helpful.
Re:Why are my Go executable files so large? by stefan_ on Sunday December 29, 2019 @11:11PM (#21910747)
External symbols have forever been the default on Windows, with the binary only containing the path to the pdb file.

Of course this is partially out of necessity; Windows is proprietary software, so they don't want to give you full debug info. But then in practice their tooling is just so vastly superior. You can fire up WinDbg, a rock stable debugger, attach to a random process and get a proper backtrace including full symbols for all the proprietary Windows stuff, because they run a public symbol server and all the symbol data you need for your Windows build is downloaded in a few seconds, fully transparently. And they ship the same symbol server with their tooling so you can run it for your own binaries, too. You can't do that on any Linux distro without starting to manually install random -dbg packages.

(And don't get me started on trying to debug crashed processes on Linux. A big reason Android has their custom libc is so that they can install default signal handlers for things like SIGSEGV, where on a normal Linux system that process goes immediately to core dump and is essentially fucked for a debugger wanting to look at its state.)

Re:Why are my Go executable files so large? by saagarjha on Monday December 30, 2019 @02:44AM (#21911549)
What’s wrong with core dumps when trying to introspect state at crash-time?
Re:Why are my Go executable files so large? by Exuma on Sunday December 29, 2019 @02:06PM (#21907745)
So what would a solution be? How do other languages solve that problem?
Re:Why are my Go executable files so large? by umanwizard on Sunday December 29, 2019 @02:20PM (#21907845)
In my experience you would strip the symbols out of the prod binary, and save them separately somewhere.

Then your production binary will give you stack traces like [0x12345, 0xabcde, ...], but you can use the separately-stored files to symbolicate them and get the source file/line info.

Not sure if this is possible on all platforms but it at least is for all combinations of {C, C++, Objective-C, Rust} and { Linux, macOS, iOS } .

And if that added operational complexity is not worth the size savings, you can freely choose not to do it, and things will work like they do in Go.

Re:Why are my Go executable files so large? by Jasper_ on Sunday December 29, 2019 @02:23PM (#21907861)
Separable debuginfo which can be loaded at runtime. DWARF uses an efficient compression mechanism much smarter than a table for this sort of mapping. And of course things like coredumps and crash dumps being sent to automated processing where devtools have the full debug symbols, while production deployments do not.

Go's insistance not just reinventing the wheel but on actively ignoring core infrastructure improvements made in the last 20 years is bizarre.

Re:Why are my Go executable files so large? by loeg on Sunday December 29, 2019 @10:04PM (#21910454)
DWARF will also just gzip sections, in addition to the compressed format.
Re:Why are my Go executable files so large? by zorked on Sunday December 29, 2019 @02:14PM (#21907801)
That is debug information. Just have it stored elsewhere (not on the binary you ship everywhere) and use that in conjunction with your core dump to debug.
Re:Why are my Go executable files so large? by asveikau on Sunday December 29, 2019 @02:17PM (#21907826)
A lot of them have symbol files separate from the binary. Unixy tooling doesn't do this by default but for example objcopy(1) in binutils can copy symbols to another file before you run strip(1), and on Mac my memory is rusty but I think it may be dsymutil(1) that lets you copy to a .dSYM bundle. Microsoft has its .pdb files and never even keeps debug info inside the binary proper.
Re:Why are my Go executable files so large? by iainmerrick on Sunday December 29, 2019 @02:23PM (#21907856)
The debug info is in a separate file. You only need that file when you’re inspecting a crash report, so it doesn’t need to be pushed out to the host device(s).
Re:Why are my Go executable files so large? by unnouinceput on Sunday December 29, 2019 @02:17PM (#21907828)
Because it's not a problem. So everybody does the same. And is not about the programming language, is about the programmer's choice. If it wants debug info inside a production program, the language let it happens. In today's age size of your executable is a non-issue. The only issue should be your performance.

Here is an example from my past. As embedded programmer I went and added manually a hundred lines of constants, which initially were just an array generated at start and increased that code by about 5%. Why? Because I gained 5 ms in execution speed. And in embedded world that's huge. Especially when your code is executed on the lowest 10 ms cycle. So the department head approved such a huge change because the code size doesn't matter, you can always buy a bigger chip, but if your car don't start because the watchdog keeps resetting your car's on-board computer then speed in code execution is everything.

Re:Why are my Go executable files so large? by saagarjha on Sunday December 29, 2019 @04:15PM (#21908538)
> Because it's not a problem. So everybody does the same. And is not about the programming language, is about the programmer's choice. If it wants debug info inside a production program, the language let it happens. In today's age size of your executable is a non-issue. The only issue should be your performance.

No.

Re:Why are my Go executable files so large? by mjcohen on Monday December 30, 2019 @12:59AM (#21911149)
Well, that settles that.
Re:Why are my Go executable files so large? by saagarjha on Monday December 30, 2019 @02:32AM (#21911505)
I figured that it would be fairly obvious why claiming that executable size means little today and that performance is the only metric that matters is a gross misrepresentation of the software industry.
Re:Why are my Go executable files so large? by mynegation on Sunday December 29, 2019 @02:17PM (#21907827)
I do not know much about go, but languages like C++ and Java give you the tools to make tradeoffs appropriate to your situation: externalizing or stripping symbols and/or debugging information.
Re:Why are my Go executable files so large? by LessDmesg on Monday December 30, 2019 @03:21AM (#21911679)
Is it just me or something like Golang shouldn't be included in production builds at all?