I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent huge pages in the background (and struggling to find/make unfragmented physical address space). The behavior change was due to the mmap size change they observed: before with the extra bytes the kernel didn't initially use huge pages. After, it did, so no background operation was necessary. (Or maybe MAP_SHARED helped because they had them backed by a non tmpfs filesystem, where huge issues aren't supported. Or maybe some behavior changes were just because they tested multiple times without rebooting and the physical page space got fragmented. Again, hard to know much from this article.)

If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction. Then I'd switch to using dedicated huge pages, because huge pages really do speed things up by like 15% for a memory heavy workload; they're worth it if you can avoid THP's sometimes pathological behavior.

Edit to add: Here is a much clearer article explaining the same problem. https://pingcap.com/blog/why-we-disable-linux-thp-feature-fo... I don't agree with the pingcap folks that you should permanently disable THP altogether but they did good debugging.

The report does look a lot like transparent hugepage defrag moving around memory randomly & system CPU spikes for no good reason.

And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space.

Literally the first thing I have to do to a Hadoop cluster is go around and turn off the defrag for the multi-threaded bits we have or system CPU goes over 20% as the system goes into workloads.

The other way you end up with a thundering herd causing permanent issues is with NUMA balancing (+ false sharing). So if you scale up a system slowly, you end up allocating in the same NUMA zone for a whole section (like a 1Gb array), which makes traversing it faster, but if this happens at the same time as another thread touching the same data, it will get scheduled over in a different zone & the kernel can do a lot of busy work with zone rebalancing as you go above 50% memory usage. This used to be a big deal when the first ccNUMA Intel boxes were coming out & mysql at Yahoo used to have so much trouble with NUMA messing up memory access speed assumptions[1].

The 96 core + 235GiB of data suggests there is some NUMA messiness going on for sure, particularly because there's a pinned worker to a CPU in the design.

The NUMA issues were front-and-center when the Power8 porting of Hadoop was going on , particularly because Java just gives you two things you can tweak directly in the GC (UseNUMA and UseTLAB). Because it was a lot of Cores on the bus (128 cores x 1TB - NUMA is like 1x-4x slower, which is hard to optimize for in a general sense).

[1] - https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-...

> And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space.

Also tmpfs, fwiw. https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... says "Currently THP only works for anonymous memory mappings and tmpfs/shmem. But in the future it can expand to other filesystems." I'm eagerly waiting for that future...

I don't think they're using tmpfs because they said "The default tmpfs on the host was untouched and was left at 50% (128 GiB)", and 235 GiB doesn't fit in 128 GiB.

Well, that's basically it.

The author appears to have heard some explanations on Unix & OS internals, but not quite understood them and appears to confuse a lot of things. From the looks of things, I figure the author did some voodoo problem solving and then, convinced of having understood the problem, decided to write an article about it.

There's a lot of second guessing, e.g. what the mmap flags do, based solely on their name, resulting in the bizarre connection to security guarantees; casually dismissing over-commit (because "nobody would do such a thing"), etc...

It's confusing because his re-write isn't multi-process, but multi-thread. But then he keeps calling threads processes.

But that is just plain wrong; it's practically in the definition of threads that they share the VM between themselves. If MAP_PRIVATE meant "copy-on-write even for threads using std::vector" most multi-threaded programs would stop working, save for perhaps a couple of purely functional examples.

On the positive side, there wouldn't be any data-races. :)

Now some of the other libc variants used by space optimized containers like alpine will still use the legacy fork()

But I constantly see developers confused because they were taught that the v7 style fork() is still used in modern POSIX.

It is didactic and not a rule, focusing on clone() helps get past the change in behavior for both threads and processes.

This author seems to have missed this change in behavior and went on a few crazy paths.

If any libc is translating fork() calls into pseudo-equivalents of "vfork" then that libc is just not-POSIX.

https://man7.org/linux/man-pages/man2/fork.2.html

"C library/kernel differences Since version 2.3.3, rather than invoking the kernel's fork() system call, the glibc fork() wrapper that is provided as part of the NPTL threading implementation invokes clone(2) with flags that provide the same effect as the traditional system call. (A call to fork() is equivalent to a call to clone(2) specifying flags as just SIGCHLD.) The glibc wrapper invokes any fork handlers that have been established using pthread_atfork(3)."

In Linux really threads are just tasks that share memory and processes are tasks that do not.

fork() and pthread_create() simply end up in the same place because user threads or processes are just tasks to the kernel.

Kernel threads, which are not user space are different.

https://github.com/torvalds/linux/blob/master/include/linux/...

The sched wouldn't be able to use a Red-Black trees in the way it does if it didn't

But yes, in the naive user perspective case the contract hasn't been violated.

The flags on that call and mostly what is inherited or dropped is the main difference.

Add in pid remapping with namespaces in containers and it is common for people to go on wild goose chases if they assume that the legacy fork() model is in play.