With glibc, you can use -ftls-model=initial-exec (or the corresponding variable attribute) to get offset-based TLS. The offset is variable per program (unlike local-exec), but the same for all threads, so it's more efficient. Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized. This is not a problem if the shared objects are loaded through dependencies at process start.

If initial-exec TLS does not work due to the dlopen issue, on x86-64 and recent-enough distributions, you can use -mtls-dialect=gnu2 to get a faster variant of __tls_get_addr that requires less register spilling. Unfortunately glibc and GCC originally did not agree on the ABI: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874 https://sourceware.org/bugzilla/show_bug.cgi?id=31372 This has been fixed for RHEL 10 (which switched to -mtls-dialect=gnu2 for x86-64 for the whole distribution, thereby exposing the ABI bug during development). As the ABI was fixed on the glibc side in dynamically-linked code, the change is backportable, but it's a bit involved because the first XSAVE-using change upstream was buggy, if I recall correctly. But the backport is definitely something you could request from your distribution.

Note that there was a previous bug in __tls_get_addr (on all architectures that use it), where the fast path was not always used after dlopen: https://sourceware.org/bugzilla/show_bug.cgi?id=19924 This bug introduced way more overhead that just saving registers. I expect that quite a few distributions have backported the fix. This breaks certain interposed mallocs due to a malloc/TLS cyclic dependency, but there is a workaround for that: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0...

The other issue is just that the C++ TLS-with-constructors design isn't that great. You can work around this in the application by using a plain pointer for TLS access, which starts out as NULL and is initialized after a null check. To free the pointer on thread exit, you can use a separate TLS variable or POSIX thread-specific data (pthread_key_create) to register a destructor, and that will only be accessed on initialized and thread exit.

This sort of question is probably more suited to libc-help: https://sourceware.org/mailman/listinfo/libc-help/

> Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized.

This feels like the perfect situation to preallocate a gigabyte or something of virtual memory for extending the TLS, similar to how the stack is. But, testing on my system, looks like the allowed initial-exec TLS size is just ~1700 bytes.

There is the idea floating around to do this for the main thread (especially for audit mode, where the automatic static TLS size discovery does not work due to the way the audit interfaces are defined, it's not merely an implementation limitation). This would allow for adjustment of initial-exec TLS expectations if the critical dlopen calls happen before the process goes multi-threaded. For threads created with pthread_create, the TCB is still at the top of the stack, which is quite efficient for access. Moving that would probably consume a TLB slot. Spacing things far apart due to aggressive address space reservations will impact TLB miss performance.

Remember that you pay this cost for every thread, not just the main thread.

Frankly, if your name isn't `libGL.so` you shouldn't even try to mix initial-exec with dlopen. Just link your libraries normally dammit!

But with virtual memory it wouldn't be much of a cost at all (..on 64-bit systems, that is; things are more sad on 32-bit if one cares about those). Just some kernel-internal data structure configuration to ensure that future memory page allocations don't overlap this one.

dlopen is a requirement for importing native libraries in non-compiled languages; and, regardless, I as a library author don't get to choose whether users will avoid using dlopen and so have to assume worst-case.

Virtual memory still costs, you know, something like 0.2% of virtual memory space in page table entries. 1 GB of VMA per thread is 2MB of real RAM cost per thread. And there's absolutely no need for that kind of space use -- the thread-local variable can just be a pointer to a heap-allocated large object.

Yeah, a gigabyte is most likely extremely overkill indeed, a megabyte or so would be plenty; though the goal would be to get threadlocals to be able to be as arbitrarily large as non-initial-exec threadlocals so it wouldn't break anything ever.

I don't know how the kernel manages it internally, but there's no need for PROT_NONE preallocated virtual memory to be mapped to actual CPU-accessible pages at least; and `mmap(NULL, 1ULL<<46, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0)` takes ~4 microseconds to map 64 terabytes of virtual memory so it's definitely not 0.002x overhead. (perhaps the overhead amount changes depending on how close to a page level the size is, but it shouldn't be too much regardless)

This'd essentially be turning the preallocated TLS space as a memory allocation arena (and you could actually even just choose to provide an alloc+free interface for programs to dynamically allocate fs-relative-offsets to use for custom threadlocals?).

(then there's general problematicness of virtual memory; such PROT_NONE never-touched memory still counts towards virtual memory usage, which is annoying; browsers/Java/etc already suffer from this, but it'd be rather ugly for literally all processes to have such. I'd quite like a memory usage counter that includes all memory that is or ever was writable, but not PROT_NONE never-touched; i.e. how much memory the process can eventually require without running explicitly requesting more via syscalls, but afaik such just doesn't exist, or at least isn't a standard-displayed thing)

> how much memory the process can eventually require without running explicitly requesting more via syscalls

This concept is called "commit charge". Windows MM models it explicitly. Linux ought to as well. I agree it's a more useful concept than just address space allocated.

Interesting! Some searching later, looks like htop's DATA/M_DRS counter (i.e. second-to-last number in /proc/<PID>/statm) appears to count something related-ish; i.e. doesn't count a PROT_NONE mmap, but does a PROT_READ|PROT_WRITE untouched one; nothing in statm appears to count untouched writable MAP_SHARED though, though (potentially?-)shared mappings do get complicated in general.

Is MADV_FREE memory charged? It contributes to classic RSS, but can be discarded by the kernel if it deems that beneficial.

Some more experimentation later, it seems to be more like just counting PROT_WRITE+MAP_PRIVATE mappings or so; i.e. mprotect(PROT_NONE)ing (or even just PROT_READ) a writable region results in it not being counted, even if the region was modified and thus must actually be persisted. So it can actually get meaningfully lower than RSS. :/

MADV_FREE doesn't affect anything afaict.

Only if mapped. If not mapped, there's no need for page table entries.

In addition to the ways that page table entries can be avoided, the system can use large pages for all the areas you aren't using yet, cutting the overhead to 4KB.

Take a look at https://maskray.me/blog/2021-02-14-all-about-thread-local-st...

I think what you want is to force the "initial exec" model using an attribute to get the more efficient code.

IIRC this (or something equivalent) is what libGL does because basically every OpenGL function call needs to read the thread-local variable holding the current GL context.

The downside is that dlopen()ing your library may fail.

Doesn't the "initial exec" model require the user to run the executable with LD_PRELOAD=your.so or linking against your.so (DT_NEEDED)? eg would it work with a Python extension module which is dlopen'd at runtime?

It depends on the amount of TLS and the number of objects that do this, and the C library. In glibc, we have a small static TLS reservation dedicated for use by dlopen of shared objects with initial-exec.

For your specific use case, CPU-local (rather than thread-local) memory might also be an option. Search for rseq on Linux. Relatively fresh feature though, not a lot of experience around it exists yet.

Known users:

- LTTng tracer

- tcmalloc

Curious if there are other prominent users of rseq.

For a tracing profiler, you want to know which thread a function call or return was made by. LTTng has kernel modules which it can use to trace context switches, and then a per-CPU trace buffer is fine, provided that you get cheap atomic writes which rseq can be used for.

Funtrace on the other hand does support ftrace for tracing context switches (https://yosefk.com/blog/profiling-in-production-with-functio...), but it doesn't require ftrace for tracing function calls made by your threads (the problem with ftrace as well LTTng's kernel modules being, of course, permissions; which shouldn't be a problem in any reasonable situation by my standard of "reasonable", but many find themselves in unreasonable situations permissions-wise, sadly.) So I don't think funtrace can use rseq, though I might be missing something.

Presumably you could store the TID in every event, or otherwise check whether the TID has changed since the last time it was logged and push a (timestamp, TID) pair if so. Reading TID should be cheap.

In what sense should reading the TID be cheap? You would need either a syscall (not cheap) or thread-local storage (the subject of TFA.) Avoiding the use of TLS by reading the TID can't really work

It looks like the TID is stored directly in the pthread struct pointed to by %fs itself, at a fixed offset which you can somewhat-hackily compile into your code. [0]

In the process of investigating this, I also realized that there's a ton of other unique-per-thread pointers accessible from that structure, most notably including the value of %fs itself (which is unfortunately unobservable afaict), the address of the TCB or TLS structures, the stack guard value, etc. Since the goal is just to have a quickly-readable unique-per-thread value, any of those should work.

Windows looks similar, but I haven't investigated as deeply.

[0] https://github.com/andikleen/glibc/blob/b0399147730d478ae451...

[1] https://github.com/andikleen/glibc/blob/b0399147730d478ae451...

This wouldn't have syntax sugar like thread_local, but presumably you're just calling some inlined helper function?

How do you avoid the problem of threads migrating between CPUs at arbitrary instruction boundaries?

> How do you avoid the problem of threads migrating between CPUs at arbitrary instruction boundaries?

rseq is short for restartable sequences; you mark beginning and end of a range of instructions, plus an abort path. The kernel checks this during task-switch and if you're within the range the instruction pointer is changed to the abort path on resumption.

(It's called restartable because the assumption is that the abort path will try again. Or at least, aborting the block of instructions midway through is recoverable.)

The primary limitation is that the "result" of the block in most cases needs to be concentrated into the last instruction of the block (similar to an atomic release write.) Otherwise you'd need to somehow recover from partially executed rseq blocks having changed some visible state but not fully completed.

Be careful, we ran into a segfault with rseq when used in a DSO loaded by another DSO.

Is using pthread_key_create / pthread_setspecific / pthread_getspecific / pthread_key_delete better or worse or the same? As I understand it, this is the low level API you were looking for.

iirc pthread uses the same ABI under the hood, only its compiled into the libc

It's much worse, since you can't inline it; it becomes a shared-library call.

Ah, but you can "inline" it by hardcoding its implementation details into your code in the most horrible way imaginable, as suggested by a crafty reader and which, scarily enough, seems to be the fastest and most robust TLS access method from a shared library: https://yosefk.com/blog/cxx-thread-local-storage-performance...

GCC and Clang both provide `_readfsbase_u64()`, which presumably is just a wrapper around `__asm__("rdfsbase %0" : "=r"(result))`. Also gcc lets you define variables like `int__seg_fs *foo;`if the `__SEG_FS` preprocessor symbol is defined. With Clang, you can use `int __attribute__((address_space(257))) *foo`

Note that FSGSBASE support arrived ridiculously late in Linux, in Linux 5.9. It was first added to CPUs with Ivy Bridge: https://www.intel.com/content/www/us/en/developer/articles/t... AMD support is of similar vintage.

As a result of the later kernel support, it may not be for everyone to turn on. Until then, the replacement on GNU Linux is just to load %fs:0, which the x86-64 ABI requires to have the same value (or %gs:0 on i386). However, usually, for initial-exec TLS access, the address of variable is not required to be in a register.

Is SEGFS/%fs-based access slower than loading the base address with RDFSBASE (which may require spilling a register) and then using base+offset access? I haven't seen such reports.

To the author: If you have a specific situation that the codegen is failing to optimize well (e.g. the ctor cases that you ran into) you can store the offset to your variable in a non-TLS global, then manually add the FS/GS to it. Use inline asm if you need to bypass any init checks.

An alternative that might be worth looking into is just hashing the FS/GS into a table index. It will be slower than the well-optimized case, but it will let you opt out of the TLS allocation process altogether. This might be a good thing in some cases for a low-level facility like a function tracer.

I have updated the post with all the suggestions people replied with except this one, because I don't understand it. How do I allocate memory for addressing it with FS/GS? Isn't FS the register pointing to the TLS area - then how is the FS-based access you propose different from how TLS normally works? Isn't GS used "for something" on x86?.. If you could elaborate on this/show some code I would be very grateful!

Note that my question is about shared libraries. If the thread_local is linked into an executable, I guess you could indeed save the offset somewhere and then add the value of %fs to it, though if this is a way to work around the constructor issue, I prefer to not have a constructor. The question is if this sort of direction can help for thread-local storage allocated by a shared library.

> How do I allocate memory for addressing it with FS/GS?

Allocate the variable normally, then compute the offset in one thread (e.g. offset = uintptr_t(&variable) - get_fs()), then access it by adding the offset to FS in any thread (e.g. (vartype *) (offset + get_fs())). The only difference from how it normally works is that you can manually force it to be inlined, sidestepping the codegen problems you described in your post. But if you can avoid those problems by not using constructors instead, that's definitely better.

I used "FS/GS" because GS is used instead of FS on some systems for the same purpose.

The shared library-specific issues are one of the reasons I was suggesting maybe looking into hashing, e.g. perhaps as a fallback solution when the TLS approach fails.

I think GP is talking about something like this [0]. You let it call __tls_get_addr() once in a constructor, take the offset from %fs, store it in a static variable, and use that offset directly. (The static variable doesn't need to be atomic, since it's only written to once, at dlopen() time.)

[0] https://godbolt.org/z/o6se3je8v

__builtin_thread_pointer() is the non-x86-specific way to get the appropriate register

Could it be productive to report the slowness to the compiler (or ld) makers? Is this inherently bad or could it be fixed by the toolmakers?

Big fan. I wish you could write more. I am gonna go check the assembly of my own code for this. And I am going to integrate funtrace into my workflow.

Question about your build system comment: my build system doesn't stoop to figuring out if `-fPIC` is needed, but it also doesn't add it unless the user asks. Were you talking about that or build systems that add it automatically?

Thanks for you kind words!

I don't think build systems add -fPIC automatically, nor remove it automatically. C++ build systems do not stoop to the question of how to best build a C++ program, by and large. They are more task graph executors - either bad ones, like make, or good ones, like Bazel, but mostly task graph executors; the most "C++ support" you will get is native support for scanning #include files (as opposed to doing it yourself like make forces you to.)

Makes sense.

I want to build a "standard library" for my build system that would stoop to that. I can do that because my build system is not just a task graph executor; it is backed by a full programming language and can add its own libraries to that language. IOW, I can add a `cpp` package to the build system that implements support for how to best build a C++ program.

So if you have a wishlist for that support, I'd love to hear it.

Idk about Bazel, but Buck2 (and Buck1, but 2 is better in every way) does handle adding -fPIC for shared libraries and leaving it off when statically linking. Each library has a PIC and non-PIC version, and the appropriate one is selected depending on if the library is being linked into an executable or a shared library. (The versions are only built when needed.) I don't remember if it also handles -fPIE similarly smartly though if you do want position-independent executables.

> I don't think build systems add -fPIC automatically, nor remove it automatically.

GNU Autotools do, if you tell, that you have a shared library. You can also switch between static/dynamic linking at build time and if you also use GNU Libtool, it will figure out the flags of your build platform at build time.

> as opposed to doing it yourself like make forces you to

You can definitely don't have to do that. GNU Automake does it by default, but if you are using plain Make, you can also use makedepend or the appropriate flags of your compiler.

> either bad ones, like make

What is wrong/missing with make as a task graph executor?

> or good ones, like Bazel

What can Bazel do better?

Hi, love the article. You mention in the article that a hardware mechanism for tracing should exist -- have you investigated the intel_pt (processor trace) extension? I believe this uses hardware buffers and supports timestamping & cycle counters (at somewhat larger than instruction granularity sadly, although it might issue forced stamps at on branches, not sure).

You can also use the PTWRITE instruction to attach metadata to the stream which seems very powerful.

Hope we can see such an extension on AMD as well.

Intel PT is indeed useful (although very, very slow compared to regular sampling profiling), but there's hardly any CPUs that actually implement PTWRITE. (IIRC there's some obscure Xeon or something?)

Typically you get a cycle count every six branches, give or take.

Sampling profilers are indeed very low-overhead, however they can't help debug tail latency, for which tracing profilers are indispensable:

https://yosefk.com/blog/profiling-in-production-with-functio...

https://danluu.com/perf-tracing/

Regarding the slowdown - magic-trace reports 2-10% slowdowns which IMO is actually fine even for production (unless this adds up to a huge dollar cost, for most people it won't) since in return for this you are actually capable to debug the rare slowdowns which are the worst part of your user experience.

However, the hardware feature that I propose (https://yosefk.com/blog/profiling-in-production-with-functio...) would likely have lower overhead since it relies on software issuing tracing instructions, eg at each function entry & exit (rather than any control flow change), and it could be variously selective (eg exclude short functions without loops; and/or you could configure the hardware to ignore short calls. BTW maybe you can with Intel Performance Trace, too, I'm just not really familiar with it.)

I discuss Intel Performance Trace in the writeup where I propose my much simpler hardware support for tracing: https://yosefk.com/blog/profiling-in-production-with-functio...

Like I said there, I'm frankly shocked that all CPUs haven't raced to implement similar features, that magic-trace which is built on top of Intel Performance Trace isn't used more widely, and that developers aren't insisting on running under magic-trace in production and requiring to deploy on Intel servers for that purpose.

The extension I propose is much simpler, and seems similar to what PTWRITE would do if it was the only feature in Intel Performance Trace. I have a lot of experience in chip architecture, and I believe that every CPU maker and every chip maker can support this easily - much more so than full feature parity with Intel Performance Trace. I hope they will!

One concern with PTWRITE is that it is somewhat "slow," at least according to this: https://community.intel.com/t5/Processors/Intel-Processor-Tr...

I wonder if this is a general issue relating to memory ordering or out-of-order execution, or whether this can be implemented more efficiently in a different extension.

Thank you for the linked article! Agreed on the huge potential for using these tools in production. The community could definitely benefit (even indirectly) by pushing for this kind of instruction set more widely.

Calling perf a sampling profiler is somewhat misleading; it can trace and count as well.