News for hackers. Stuff that matters.
Re:I wrote a linker everyone can understand by barrkel on Wednesday June 09, 2021 @06:33AM (#27446341)
Smart linkers (and most linkers these days are smart linkers) are a lot like tracing, compacting garbage collectors.

A GC has a series of roots which point at blobs of memory. The GC needs to understand where further references are located in the pointed memory; a simple GC follows these references, avoiding cycles, until it has traced through the live set. It calculates new addresses for each blob of memory, then goes through the live set a second time, moving each object into its final position and adjusting all the references to point to their final locations.

A linker starts out with a main() function or equivalent which is a blob of code, contained in an object file produced by the compiler. The object file contains a table of references to all the external symbols used by the code, as well as the locations in the code that the references are made, classified by type of reference (e.g. absolute vs relative references). The references in the main() function are like the set of GC roots. The object file also contains a table of exports, all the symbols which it exposes externally.

The linker then locates the object code or data for each unresolved external reference (in other object files supplied on the command line, or in archive library files - indexed concatenations of object files). This process is followed recursively until the live set - the set of blobs of code and data - to be assembled into the final executable is determined.

Once all these blobs have been found, they can be copied into the final executable, and all the references to symbols can be filled in with appropriate offsets. There's details here: relative offsets may be baked in, but the locations of absolute offsets need to be marked in the final executable so that the operating system loader can adjust them if it needs to load the executable somewhere other than its preferred address.

The operating system loader is itself a linker. It, too, needs to follow a similar process, but instead of object files, it's dealing with shared objects (DLLs in Windows), and it's chasing down external references, loading things, and adjusting offsets as necessary (address space layout randomization (ASLR), load address conflicts (a lot more common in 32-bit address spaces)). The OS loader can also be lazy; instead of resolving a reference, it can fill it with a stub which is only resolved on first call. This delays linkage failures, but if the application tests for versions etc. before invoking, the developer gets the ergonomics of early bound linking with the features of late-bound linking that would otherwise need manual use of dlsym / GetProcAddress.

If you're interested in the topic, I highly recommend Linkers and Loaders by John Levine, long-time moderator of comp.compilers.

Re:I wrote a linker everyone can understand by juloo on Wednesday June 09, 2021 @06:45AM (#27446406)
GC traverses graph, linker traverses graph. GC = linker. I disagree. Also if that's the description of a "smart linker", what is a "dumb linker" then?
Re:I wrote a linker everyone can understand by barrkel on Wednesday June 09, 2021 @08:54AM (#27447324)
Not just traversing a graph, but determining the location of blobs of data, and adjusting offsets inside those blobs of data to point at the relocated blobs of data. If your GC is simple, there's almost a 1:1 correspondence with every write to memory.

Smart linker relates to eliminating dead code. Depending on how symbols are packaged, or how well the compiler is integrated with the linker, there's a risk of including private (i.e. non-exported) code/data in the final executable which is never referenced but is simply co-located with code or data which is linked in. But it's not really a technical term, it's a marketing term from Delphi.

Re:I wrote a linker everyone can understand by chrisseaton on Wednesday June 09, 2021 @09:02AM (#27447377)
> But it's not really a technical term, it's a marketing term from Delphi.

Like 'optimising compiler' - are there really any non-niche compilers that do not optimise? Maybe some first-tier JITs.

Re:I wrote a linker everyone can understand by barrkel on Wednesday June 09, 2021 @09:24AM (#27447572)
Well, if the linker is very dumb, and just wholesale includes everything in an object file if there's any reference to it, the optimizing compiler can try and work around that by producing itty bitty object files, one for every symbol.

Linkers have more scope for smartness. The size of a block of code can be quite sensitive to the addressing mode (near vs far for e.g. 16-bit x86, relative offsets of various widths vs absolute etc.) used for each reference. An intelligent linker can adjust the code depending on where the target ends up, altering the addressing mode. The scope for optimization here depends on the ordering in the final executable, density of references and so on.

Re:I wrote a linker everyone can understand by Joker_vD on Wednesday June 09, 2021 @09:59AM (#27447889)
But doesn't this "smartness" require accurate information about internal symbol usage? For exmaple:

    static int foo(void) {
        return 4;
    }

    int bar(void) {
        return 42 + foo();
    }
The compiler, I believe, may emit no symbol entry for foo() and use a relative call inside bar() to call it, so if you only include "used" symbols in your resulting object file, bar() will end up calling into who knows where. That's why there are special flags for enabling "function-level linking" and all that jazz, right?
Re:I wrote a linker everyone can understand by barrkel on Wednesday June 09, 2021 @12:28PM (#27449854)
And this is where Delphi's smart linker is smart: it's always using function-level linking.

(I can't answer your question in detail because it's been a long time since I've been poking around in object files and I don't know what C/C++ compilers and linkers have been doing with them recently. I was a compiler engineer at Borland for more than 5 years, but it was years ago.)

Re:I wrote a linker everyone can understand by megameter on Wednesday June 09, 2021 @08:33AM (#27447163)
The kind where the user has to chase dependencies.
[dead]
Re:I wrote a linker everyone can understand by chrisseaton on Wednesday June 09, 2021 @08:49AM (#27447287)
> I felt enough rage about your comment

This isn't a normal response to a comment on the internet.