News for hackers. Stuff that matters.
Re:Sta.li: Static Linux by davidp on Tuesday February 18, 2014 @11:45PM (#7261846)
I'm puzzled by the idea of a system being leaner/faster with n copies of a library in physical RAM rather than 1 copy mapped via VMM into whatever process wants it. IIRC this was the main point of shared libraries, not pluggability or changing code during runtime. Am I missing something?
Re:Sta.li: Static Linux by cgh on Wednesday February 19, 2014 @12:20AM (#7261953)
Static linking doesn't necessarily link the entire library, unless the entire thing compiles to a single .o file. Linkers are smart enough to only link in the object files needed by the program. So assuming you are only linking in well-designed libraries, I guess it's possible that statically-linked software will be smaller since it will leave out the stuff you aren't using.
Re:Sta.li: Static Linux by spc476 on Wednesday February 19, 2014 @01:56AM (#7262241)
I recently played around with this, taking a rather small project (around 15,000 lines of code), putting it all into a single file and compiling. It did produce a smaller executable, but the real gain was in making every function static (since it's all in a single file). Doing that, a total of 41 functions were eliminated (either inlined or not used at all).

Was it worth the effort? Eh. But it was instructive and I'd like to attempt (when I get some time) to try a larger project.

Re:Sta.li: Static Linux by exDM69 on Wednesday February 19, 2014 @03:43AM (#7262561)
Some popular software like SQLite combine all their sources into one big source file called Amalgamation and then compile that. Their benchmarks show modest but not negligible performance gain.

There's a lot of work going on in link time optimization at the moment, both in LLVM and GCC. It's not quite ready for prime time, it still takes more than a small change in your Makefile to deploy it (e.g. dealing with linkers etc).

With LLVM toolchain you can compile C code (or other high level code) into LLVM IR, link the IR files together and run that through the optimizer.

You will notice that modern optimizers will want to inline everything if possible and a lot of functions will be missing from the resulting binary. Boundaries of object files are perhaps the biggest obstacle in optimization today.

Re:Sta.li: Static Linux by alexchamberlain on Wednesday February 19, 2014 @03:15AM (#7262441)
You essentially did Whole Program Optimisation by hand.
Re:Sta.li: Static Linux by mpyne on Wednesday February 19, 2014 @12:06PM (#7264603)
We did that (as a developer option) with KDE as well, since KDE 2 or thereabouts?

With the automake-based build system you'd pass "--enable-final" and the buildsystem would cat all the source files together and compile the whole damn thing at once (and really stress-test the kernel and gcc).

With KDE 4 I believe it is -DKDE4_ENABLE_FINAL=TRUE passed to cmake.

It was never quite 100%... sometimes you'd run into things like different source files in a modules declaring the same class name, insufficiently-namespaced header include guards, etc. But it was definitely interesting.

Re:Sta.li: Static Linux by Jare on Wednesday February 19, 2014 @11:05AM (#7264228)
That approach is common practice when developing for game consoles.
Re:Sta.li: Static Linux by pjmlp on Wednesday February 19, 2014 @01:56AM (#7262243)
> So assuming you are only linking in well-designed libraries

This is a very big assumption.

Dynamic linkers are also clever enough to only mmap the required parts of dynamic library.

Re:Sta.li: Static Linux by npsimons on Wednesday February 19, 2014 @10:16AM (#7263997)
It's been awhile since I've mucked about with these sorts of things, but I'm glad that my thought about that is confirmed: if static linking only brings in used functions, why doesn't dynamic loading do the same (it does, apparently)? Much of this railing against dynamic loading wasting resources seems like complaining about the wrong things, either bad dynamic linkers, or bad libraries, neither of which will be fixed by static linking.

Don't get me wrong, there are places I think that static linking is ideal. I wish more distributors of binary only software would statically link, or at least include standalone required dynamic libraries, rather than rely on system dynamic libraries.

I wish them luck in their experiment and hope they can improve static linking, but I suspect they will learn more about why dynamic loading "wastes" so many resources the more they come in contact with real world libraries.

Re:Sta.li: Static Linux by mpyne on Wednesday February 19, 2014 @12:09PM (#7264617)
AFAIK static linking doesn't bring in "used functions".

It brings in used libraries, all at once. E.g. if you used sincos() from math.a, and math.a contained 47 other math functions, then you'd get all 48 math functions in your static binary just from using sincos().

Someone correct me if I'm wrong but I believe it's only with good whole program optimization at link time that it's possible to truly prove that a function is unneeded and exclude it (and then re-link if needed to re-resolve symbols to their new address in virtual memory).

Re:Sta.li: Static Linux by alexchamberlain on Thursday February 20, 2014 @03:52AM (#7269712)
You are wrong... sort of. It brings in used objects. A library can be made up of many objects; unused ones will be discarded.
Re:Sta.li: Static Linux by mpyne on Thursday February 20, 2014 @09:35PM (#7275108)
Ah, good point, thanks for the correction.
Re:Sta.li: Static Linux by pjmlp on Wednesday February 19, 2014 @11:54AM (#7264528)
I don't know that well how Linux dynamic loader works, my comment was based on what is possible in other operating systems in general, and what has been done in operating system research.
Re:Sta.li: Static Linux by ithkuil on Wednesday February 19, 2014 @11:27AM (#7264357)
the whole library gets mmapped, there is no point doing otherwise, the mapping itself is cheap.

You are probably talking about demand paging (which happens on statically linked binaries.

Re:Sta.li: Static Linux by pjmlp on Wednesday February 19, 2014 @11:53AM (#7264519)
You probably talking about Linux, there are other types of dynamic loaders out there.
Re:Sta.li: Static Linux by ithkuil on Wednesday February 19, 2014 @03:40PM (#7266082)
Ah, interesting, do you have some links?
Re:Sta.li: Static Linux by simcop2387 on Wednesday February 19, 2014 @12:58AM (#7262072)
Combined with modern compilers and link-time optimizations they can even leave out code when it's just one .o file also. Dead code elimination ends up great with that there.
Re:Sta.li: Static Linux by clord on Wednesday February 19, 2014 @01:42AM (#7262200)
On top of what others have said, it takes some time to dynamically load a library into an address space. There are tables that may need to be walked and updated with correct pointers. For large libraries, this can be quite measurable. A statically linked executable will be memory-mapped and then brought in lazily as the program runs. And then if executed again, everything is mapped and loaded, so there is zero delay. Compare to dynamic loading, which will require table updates again (especially with ASLR).

This is why I like to re-make my shell and associated tools static binaries — shell scripts run 10-20% faster. (lots of small programs running repeatedly)

Re:Sta.li: Static Linux by davidp on Wednesday February 19, 2014 @01:57AM (#7262251)
> it takes some time to dynamically load a library into an address space. There are tables that may need to be walked and updated with correct pointers

True, but I'd expect that to be dwarfed by the I/O time required to load even a single 4k page from disk, vs. keeping one copy of a big dynamic library like glibc loaded for the whole system, with fixups done per-process.

Good points about ASLR and static-linking frequently-exec'd-and-exited processes like the shell; and certainly for embedded and HPC it makes sense. I guess the moral, as always, is to measure.

Re:Sta.li: Static Linux by prewett on Wednesday February 19, 2014 @03:43AM (#7262560)
A lot of those pages will already be in the file cache (assuming his use case of small utilities running frequently). Anyway, it should be easy to test: since glibc will always be in memory, any differences in timing between the static and dynamic version should be those alleged loading costs.
Re:Sta.li: Static Linux by pjmlp on Wednesday February 19, 2014 @01:57AM (#7262250)
How many minutes are saved by 10-20% faster? A few seconds don't matter for a human.
Re:Sta.li: Static Linux by clord on Wednesday February 19, 2014 @10:08AM (#7263948)
We have some scripts at work that take 10+ hours to execute. Granted most of that time is in large child processes.
Re:Sta.li: Static Linux by cwzwarich on Wednesday February 19, 2014 @02:31AM (#7262326)
They get a speedup from eliminating the indirection used for calls across dylibs. From the sound of it they're also eliminating position-independent code, which itself can be a reasonable speedup, especially on 32-bit x86.
Re:Sta.li: Static Linux by mcot2 on Tuesday February 18, 2014 @11:53PM (#7261869)
Maybe if we didn't have gigabytes of RAM these days. I quite like the idea of static linking. It makes software packaging and distribution very very easy and it has some security benefits. Go's build system is a good example of this. This project seems to be dead.
Re:Sta.li: Static Linux by weland on Wednesday February 19, 2014 @03:57AM (#7262615)
I have mixed feelings about the security gains.

On one hand, you eliminate one attack vector since you take ldd out of the equation. On the other hand, you depend on packagers who distribute their programs to rebuild and relink them every time a security issue creeps up a library they link with. I'm not sure I like that, and I don't have the free time I had in high school when compiling everything by hand seemed really fucking cool.

Re:Sta.li: Static Linux by girvo on Wednesday February 19, 2014 @01:08AM (#7262096)
It's the distribution and packaging parts that I love about it. Also might help with cross-platform distribution as well; While dynamic linking does have advantages, I get really frustrated when an old application won't run on a new kernel due to requiring old libraries that simply can't be installed. Static linking fixes that, and it's why anything I try and write is statically linked for the most part!
Re:Sta.li: Static Linux by nicolast on Wednesday February 19, 2014 @02:25AM (#7262314)
Ever looked at Nix and NixOS?
Re:Sta.li: Static Linux by eekee on Wednesday February 19, 2014 @04:52AM (#7262809)
i don't know about leaner, but static linking is faster. symbol resolution is a heck of a task to be run on every program load, especially for larger programs. i remember before linux optimized its dynamic linking, starting a gnome or kde program was a glacial process.

as for static linking being leaner, i have been told an entire shared library needs to be loaded if so much as 1 program needs a part of it, but i doubt it. i don't see why shared libraries can't be demand loaded just like executables are. then again, demand loading a shared library would be a more complex task, and i have reservations about complexity just like the suckless community does.

Re:Sta.li: Static Linux by anabis on Wednesday February 19, 2014 @12:48AM (#7262040)
In generic distributions it would cause bloat, but in a single purpose embedded device, I found it was a significant size reduction.
Re:Sta.li: Static Linux by qznc on Wednesday February 19, 2014 @05:18AM (#7262883)
If each copy uses less then 1/n of the library and the rest can be stripped away by the compiler, you need less memory.