man8alexd

I think Kahneman acknowledged problems with one chapter about priming, not the entire book.

Powered by Movable Type.

I have already pointed you to /proc/pressure/memory. Prometheus node_exporter tracks it as node_pressure_memory_waiting_seconds_total and node_pressure_memory_stalled_seconds_total. You'll see them spiking when you experience stalling.

You can also track variables from /proc/vmstat exported as node_vmstat_allocstall_(normal|dma32|movable), node_vmstat_pgscan_direct, node_vmstat_pgsteal_direct - you'll see them spike when the system enters direct reclaim.

Install and configure systemd-oomd and track its log - it will activate on memory pressure.

To track code page eviction/re-faults specifically, you'll need to write some eBPF code, there are no direct measurements for this available.

You seem to confuse user-space memory allocations (malloc) with the kernel memory page allocation and mapping. The processes don't need to allocate any new memory with malloc. They get stuck when accessing memory that is not mapped to a physical memory page or when they enter a syscall that needs in-kernel memory allocation.

Another important caveat: people keep assuming that page cache can be easily discarded and it doesn't count towards the working memory set. Then they get surprised by unexpected OOMs. There is a decade-old issue on Kubernetes https://github.com/kubernetes/kubernetes/issues/43916

The assumption that a system gets unresponsive due to code pages being reclaimed doesn't make any sense. It is a common myth, cargo-cult knowledge. The kernel doesn't discard any active referenced pages at all, and the active code pages are the last that are getting reclaimed.

The proper explanation is relatively straightforward. Normally, any memory request for a process is served by getting a page from the free memory pool. But when the system free memory goes below the low watermark (calculated from vm.min_free_kbytes and vm.watermark_scale_factor), the kernel stops giving the pages from the free memory pool and instead goes into direct reclaim, trying to find unused pages. While the kernel does reclaim, the allocating process remains stalled. The kernel can't return control to the process - there is no page mapped that the process tried to access. This applies to any types of pages - anon, file-backed, code pages. So, under the condition of free memory below the low watermark, any process trying to allocate memory gets stalled. This is measured as memory pressure stalls (PSI) in /proc/pressure/memory.

The problem with the system being stuck in an unresponsive state is that the design choice for the kernel OOM handling is to be optimistic and keep the workload running as long as the kernel is making any progress reclaiming pages. The direct reclaim process needs to fail to make any progress 16 times before the OOM killer is invoked.

The solution is to use cgroup memory limits to avoid getting into the situation of memory exhaustion and use user-space oomd to recover from this situation earlier.

Value 60 is not an arbitrary number. It makes the kernel reclaim 140 file pages for each 60 swapped-out anon pages. This proportion is supposed to reflect the relative performance cost between sequential file access and random swap I/O on HDD. For SSDs, the value 100 is more appropriate.

> I don't see a reason why the default shouldn't be 0 then on modern systems.

The zero value makes reclaim very unbalanced and is not appropriate for generic systems. This will cause the kernel to discard all the page cache, slowing down the system with uncached I/O, while keeping all unused anonymous pages in memory.

What do you think this value of 60 means? Also, as I said, in extreme situations, when the system is right before invoking the OOM killer, swap is used regardless of `vm.swappiness` setting.

In simple cases, instead of using `mlock` just set cgroup memory.min

Active code pages with the "Accessed" bit set are specifically excluded from being moved to inactive LRU when active LRU is being shrunk. Instead, they are moved to the head of the active LRU. All other pages are moved from active LRU to the inactive LRU unconditionally, even if they have the "Accessed" bit set. So, no, the active code pages are not going to be eventually evicted.

I'll try again. Why do you think that unresponsiveness is caused by code pages being discarded? Do you have evidence for this? Have you considered that there are other mechanisms that cause programs to become unresponsive under memory pressure?

All code pages in the active file LRU are excluded from the reclaim.

The assumption that data pages can always be easily discarded is incorrect. Dirty data pages can't be discarded until they are written back. Recently accessed data pages are also not discarded until all inactive data and code pages are discarded.

Could you clarify what exactly is "not true"? Also, could you explain what exactly your evidence proves? What do you think happens when the system becomes unresponsive under memory pressure, and why?

vm.swappiness doesn’t work the way most people assume it does. and it is ignored under heavy memory pressure.

You are missing that the file LRU is not only the code pages or memory mapped files but all file data. The active code pages are specifically excluded from being discarded during the reclaim process.

Location: Belgrade, Serbia (UTC+2)

Remote: yes (working exclusively remotely for the past 20 years)

Willing to relocate: yes

Technologies: AWS, Kubernetes, Nomad, Docker, podman, Hashicorp Consul, Vault, Packer, Terraform, Ansible, Chef, Puppet, Grafana, Loki, Prometheus, ELK stack, Jenkins, GitHub Actions, Flux, GitLab, Postfix, HAProxy, MySQL, MariaDB, PostgreSQL, Ruby, Python, bash, C

Résumé/CV: https://alexeydemidov.com/cv

Email: in the resume

GitHub: https://github.com/AlexeyDemidov

LinkedIn: https://www.linkedin.com/in/alekseydemidov/

ServerFault: https://serverfault.com/users/23022/alexd

Have been playing with Linux and FreeBSD for over two decades. Ran an ISP engineering team for a decade, now building and maintaining cloud infras.

Location: Belgrade, Serbia (UTC+2)

Remote: yes (working exclusively remotely for the past 20 years)

Willing to relocate: yes

Technologies: AWS, Kubernetes, Nomad, Docker, podman, Hashicorp Consul, Vault, Packer, Terraform, Ansible, Chef, Puppet, Grafana, Loki, Prometheus, ELK stack, Jenkins, GitHub Actions, Flux, GitLab, Postfix, HAProxy, MySQL, MariaDB, PostgreSQL, Ruby, Python, bash, C

Résumé/CV: https://alexeydemidov.com/cv

Email: [email protected]

GitHub: https://github.com/AlexeyDemidov

LinkedIn: https://www.linkedin.com/in/alekseydemidov/

ServerFault: https://serverfault.com/users/23022/alexd

Have been playing with Linux and FreeBSD for over two decades. Run an ISP engineering team for a decade, now building and maintaining cloud infras.

Location: Belgrade, Serbia (UTC+2)

Remote: yes (working exclusively remotely for the past 20 years)

Willing to relocate: yes

Technologies: AWS, Kubernetes, Nomad, Docker, podman, HashiCorp Consul, Vault, Packer, Terraform, Ansible, Chef, Puppet, Grafana, Loki, Prometheus, ELK stack, Jenkins, GitHub Actions, Flux, GitLab, Postfix, HAProxy, MySQL, MariaDB, PostgreSQL, Ruby, Python, bash, C

Résumé/CV: https://alexeydemidov.com/cv

Email: in the resume

GitHub: https://github.com/AlexeyDemidov

LinkedIn: https://www.linkedin.com/in/alekseydemidov/

ServerFault: https://serverfault.com/users/23022/alexd

Have been playing with Linux and FreeBSD for over two decades. Run an ISP engineering team for a decade, now building and maintaining cloud infras.

Why do you think it is a better default? What does it solve?

You have a lot of unused anonymous memory that is better swapped out and used as a file cache. Specifically for Linux swap, see https://chrisdown.name/2018/01/02/in-defence-of-swap.html

You can't fix the fact that you can't predict the future. The software will always allocate more memory than it needs because it can't predict its future resource usage, so limiting memory allocation with strict overcommit is meaningless.

About shared memory included in the memory size calculations https://lkml.iu.edu/1902.2/05674.html

The kernel keeps track of active/inactive pages by scanning page tables during the reclaim process. It only scans until it finds enough inactive pages to reclaim. Scanning all pages all the time is very expensive from a performance point of view.

You can't know when the OOM killer is awake and hunting. The kernel doesn't know that.

OOM is better. If a program doesn't handle ENOMEM properly, then its state is unpredictable and can lead to data corruption.

Linux MemAvailable from /proc/meminfo is just an estimation calculated as an arbitrary percentage (50%) of free and potentially reclaimable memory.

You can't determine how much of the memory can actually be reclaimed under memory pressure until you try to reclaim it.

There is no point in managing memory allocations as they have little relation to actual memory usage. There are also other methods than the OOM killer to handle OOM, like process throttling using cgroups "memory.high" limits.

In this specific case, the correct behaviour would be to drop a part of the buffer pool until the memory pressure is gone. The context-dependent question is how much and how fast to drop. The current implementation drops to a single configurable level but I suspect it could have implemented better heuristics.

Because no one can predict the future, and they don't know how many resources they will need.