News for hackers. Stuff that matters.
Re:The Linux kernel's inability to gracefully handle low mem... by throwaway2048 on Monday August 05, 2019 @09:15PM (#20621102)
Yep, even with no swap whatsoever, performance is completely trashed (talking even the mouse lags for 30+ seconds at a time) for like a solid 5+ minutes before the oomkiller triggers, with swap, you might as well just reboot because the system will take perhaps an hour to start responding.

Linux is completely useless with ram that is almost full in a way that OSX and windows absolutely are not.

Re:The Linux kernel's inability to gracefully handle low mem... by blattimwind on Monday August 05, 2019 @09:21PM (#20621139)
Linux absolutely, positively, requires a decent chunk of free memory because the kernel's algorithms simply do not work and in my 15+ years of Linux experience they NEVER worked. If Linux starts thrashing, and it does so easily, then the box takes what surely is an uncomputable amount of time to recover.

One way to make relatively sure that this is always the case is to use a userspace daemon like earlyoom.

Though to stay fair all desktop OS'es behave badly when put under memory pressure, it's just that Linux is an order of magnitude or so worse.

Re:The Linux kernel's inability to gracefully handle low mem... by maximente on Monday August 05, 2019 @09:37PM (#20621226)
are any of the BSDs better in your experience?
Re:The Linux kernel's inability to gracefully handle low mem... by myrandomcomment on Tuesday August 06, 2019 @12:10AM (#20622048)
Well my memory for BSD is not fully clear, but when I looked at this back in 2008 BSD handled it better then Linux. The king of sorting this was Solaris. It was rock solid. I would still argue that Solaris is better then Linux as a server, but it does not matter as Linux won. Also, I want my Amgia back :)
Re:The Linux kernel's inability to gracefully handle low mem... by ComputerGuru on Tuesday August 06, 2019 @01:47AM (#20622493)
I only ditched the Solaris train for FreeBSD when all hope was gone after Schwartz’s reign came to an end. Solaris got so many things right architecturally that are damn hard to shoehorn in after the fact. Until today, I don’t know if any other *nix has sorted out ABI compatibility across architectures, which allowed running applications cross-compiled for another target to run against the memory-resident kernel without virtualization (only instruction emulation when needed). In practice, it meant that you could run “universal” x86 binaries against both the x86 and x86_64 kernels, even calling into drivers (!!), with guaranteed compatibility. The last I had checked, FreeBSD had made it a goal to unify all numeric values of IOCTLs across architectures, but I don’t believe they are there yet.
Re:The Linux kernel's inability to gracefully handle low mem... by floatboth on Tuesday August 06, 2019 @06:14AM (#20623845)
Well, a 32-bit Linux binary on 64-bit FreeBSD can call into the GPU driver and render 3D, at least :) But I don't think anything is guaranteed for sure. 32-bit crap is not exactly a priority, haha.

On the other hand, syscalls across architectures of the same bitness are exactly the same, minus the newer architectures just not having some historical abominations like sbrk. (IIRC, syscalls are different between even aarch64 and amd64 on Linux, which is just… why?!?)

Re:The Linux kernel's inability to gracefully handle low mem... by ComputerGuru on Tuesday August 06, 2019 @08:35PM (#20631024)
Hey, pretty sure I recognize your name from other FreeBSD-related comments you've made on here in the past that I found worthy of saving!

We are stuck with needing to support x86 and prefer to do so as seamlessly as possible for the end user (call us old school like that); to that end we have a single bootable ISO image containing our appliance that needs to run on generic x86 hardware ranging from cutting edge to 15+ years old (I've posted about that with tons of accolades plus patches maximizing compatibility to the FreeBSD dev ML [some|a long] time ago but never heard back). We ported the software over from a very messy Linux-based implementation to a much cleaner architecture under FreeBSD 9 what I now realize can be considered a pretty long time ago.

With that background out of the way, we had written a custom scfb-powered "guaranteed compatible" X display driver (back before there was an upstream scfb driver, although I am sorry to see even with FreeBSD 12.0 that xf86-video-scfb does not work on quite a number of hardware profiles) that would work so long as the framebuffer did (and it usually does). I remember running into an issue (under 10.0) where even basic FBIO_* ioctl defines had different values when compiled for AMD64 vs i686, so although our x86 Xorg module would run under x64 (with the i386 compatibility module loaded, iirc?), it would fail to work as the wrong driver routine was being called into. I was honestly surprised it didn't work oob, as these were very basic ioctl calls.

Of course today with UEFI, signed drivers, and everything else, it's a miracle we haven't already been forced to distribute separate ISOs, but that's really only because we gave up on signed booting and simply require a legacy CSM for UEFI hardware, which is more steps but generally universally available; I don't think there's any real impetus behind getting UEFI support for x86, so we still boot the old-fashioned way.

Getting completely off topic, honestly both Linux and FreeBSD are quite literally decades behind when it comes to being able to guarantee unaccelerated video output at {,e,s,x}vga resolutions without requiring hardware-specific drivers; I don't think I have ever seen a MS-DOS/Windows PC that wouldn't at least boot in VGA mode going back at least all the way to early 90s. We certainly spent no time agonizing over getting video to work when Microsoft used to allow us to license WinPE to power our appliance images. I have an unfinished patch somewhere for the vt VGA driver that adds support for the various ioctls, but it seems the world has unfortunately decided that going with hardware-specific (k)drm drivers is the way forward. We decided to bite the bullet and now ship with those too, but it's about the farthest thing from bulletproof compatibility and neither the vendors nor the reverse-engineered community driver devs really bother formally testing these modules and it unfortunately very much shows. And now nVidia has discontinued providing proprietary x86 drivers for FreeBSD and Linux (although they notably still do for Solaris, almost certainly because of the aforementioned ABI compatibility guarantees there), so we're probably going to need to split our legacy/i686 and the uefi/x64 images soon enough.

Re:The Linux kernel's inability to gracefully handle low mem... by floatboth on Wednesday August 07, 2019 @04:16PM (#20638459)
Haha, that's like an epic combo of things I dislike — x86, legacy BIOS, 32-bit OS on 64-bit CPU, unaccelerated graphics, and X :)

> signed booting

It's not like FreeBSD was an early adopter of that. Some kind of secure boot support things have landed in the last couple months, and veriexec for signed stuff, but very few people have started using this.

> separate ISOs

Or a "fat" ISO that has both 32 and 64 bit versions?

> nVidia proprietary drivers

With these, there's already an issue with which versions of that support which cards..

Re:The Linux kernel's inability to gracefully handle low mem... by account42 on Tuesday August 06, 2019 @08:41AM (#20624573)
> Until today, I don’t know if any other *nix has sorted out ABI compatibility across architectures, which allowed running applications cross-compiled for another target to run against the memory-resident kernel without virtualization (only instruction emulation when needed).

QEMU supports user-space emulation on Linux, for example you can run an ARM binary on top of a amd64 kernel. x86 on top of amd64 kernels works OOTB.

Re:The Linux kernel's inability to gracefully handle low mem... by ComputerGuru on Tuesday August 06, 2019 @08:43PM (#20631043)
Interesting, thanks for sharing that info! I imagine that is restricted to running code that doesn't really make any calls against the hardware layer much?

Presumably it must also include some sort of compatibility layer rather than merely translating machine instructions, as the syscall payloads vary considerably. But qemu is certainly small and (when used correctly) fast enough that depending on that should not be a dealbreaker if you really have no better options.

Re:The Linux kernel's inability to gracefully handle low mem... by Annatar on Tuesday August 06, 2019 @02:35AM (#20622745)
I just ordered my capacitor kit for re-capping my Amigas.

Anyway Linux won for now, but Solaris lives on vicariously through illumos and SmartOS, which handle this the same way as Solaris does (it's the same code) and the codebase is very actively developed, so it's not over yet no matter how much Linux is winning now: I'm waiting for the folks to figure out just how shitty Linux is under the hood and the fact this is a topic on HN tells me it's starting to happen as people gain more and more experience. Linux won the battle but with such shitty code underneath winning the war as history will judge it is another matter entirely. Meanwhile SmartOS continues to be developed and works correctly in these situations.

Re:The Linux kernel's inability to gracefully handle low mem... by toast0 on Tuesday August 06, 2019 @02:12AM (#20622614)
FreeBSD is probably better than Linux at this, but that doesn't necessarily mean it's good.

Prior to FreeBSD 10, I'm paraphrasing but essentially it would pretty much not swap anonymous pages until memory pressure got really high -- it would evict clean disk pages and try to clean dirty disk pages first. But if the pressure continued, it would start marking anonymous pages and eventually swap those out too. A big problem was there was a massive slowdown when it hit the threshold; tons of pages would be marked as inactive, so using them in the process triggers a page fault (and they then get marked active), and all the page faults and page table activity slows things way down.

In FreeBSD 10, a patch from NetApp changed the page scanning behavior so it was always running. This is good, because you don't get that massive slowdown while the pages are determined to be active or inactive; instead the kernel always has a much better idea of which pages are used; but it's also not great because some of the inactive anonymous pages are going to get paged out, and it also makes memory accounting trickier. In FreeBSD 9, you could look at the active memory stat on a system that wasn't at the brink and figure that was pretty much the amount of ram you needed -- so if it was growing over time, you might have a memory leak, or real growth. In FreeBSD 10, it's harder, because your programs may need things loaded but not access them for some time, and the pages could get marked inactive, so you really have to guess; and swap usage isn't a great metric either because those idle pages might get swapped out. You can set the vm.defer_swapspace_pageouts sysctl to get something similar to old swap behavior, but you still lost the memory usage gauge.

OOM behavior seems better, but also not necessarily great. Most of my experience is from running Erlang servers with very little other stuff going on, so it might not apply to a more conventional server with lots of random things, and probably not to a desktop either. Overcommit is disabled by default (and i think it may be substantially different than Linux overcommit anyway), and mostly what happens in my experience is the big allocator (beam.smp for us) gets its allocations failed when you run out of swap and then it crashes and everything is in an ok state. Every once in a while, something else will get failed allocates and die before the big allocator, but then the big allocator still dies; this wasn't great because sometimes it would be an important but small daemon like ntpd, when this happened we'd often reboot to get back to a known good state. There's also an OOM killer that can be triggered depending on exactly how things go down -- it just kills the biggest process; for us, that's the right thing, if the VM is eating all the ram, killing off gettys isn't going to do any good.

However --- we saw several times that the kernel would just hang. Unfortunately, when we were seeing this often, I hadn't figured out the kernel debugger, and it was usually in the middle of an incident anyway, so we'd reboot and go on with life. A hung kernel is better than a thrashing kernel, but only a little bit.

Re:The Linux kernel's inability to gracefully handle low mem... by floatboth on Tuesday August 06, 2019 @06:32AM (#20623919)
> Overcommit is disabled by default

No way. Processes allocating terabytes (GHC Haskell compiled programs, AddressSanitizer, etc) work fine out of the box.

Reading tuning(7):

> Setting bit 0 of the vm.overcommit sysctl causes the virtual memory system to return failure to the process when allocation of memory causes vm.swap_reserved to exceed vm.swap_total. Bit 1 of the sysctl enforces RLIMIT_SWAP limit (see getrlimit(2)). Root is exempt from this limit. Bit 2 allows to count most of the physical memory as allocatable, except wired and free reserved pages (accounted by vm.stats.vm.v_free_target and vm.stats.vm.v_wire_count sysctls, respectively).

Soooo vm.overcommit=0 does not mean "no overcommit" (which would make sense), no, it seems to mean something like "no special flags like disabling overcommit" :D

Re:The Linux kernel's inability to gracefully handle low mem... by toast0 on Tuesday August 06, 2019 @12:27PM (#20626686)
You are totally right. I've failed at reading quite a bit on this one.
Re:The Linux kernel's inability to gracefully handle low mem... by mort96 on Monday August 05, 2019 @09:20PM (#20621130)
Yeah, any time I end up in that situation I just hold the power button and prey to the journaling gods. It's a serious issue, I'm extremely glad it looks like there's actually some progress on fixing it.
Re:The Linux kernel's inability to gracefully handle low mem... by minitech on Tuesday August 06, 2019 @05:30AM (#20623647)
Enable SysRq and use SysRq+F. https://news.ycombinator.com/item?id=16561746
Re:The Linux kernel's inability to gracefully handle low mem... by brendangregg on Monday August 05, 2019 @09:49PM (#20621297)
I don't think that's a fair comparison: Do you normally run OSX with the swap files completely disabled, or Windows with the pagefile completely disabled? That's what this bug is describing. I'd bet things get pretty nasty on OSX and Windows too, if you tried that.

Perhaps the real bug is that Linux distros make it easy to run swapless.

Re:The Linux kernel's inability to gracefully handle low mem... by throwaway2048 on Monday August 05, 2019 @09:54PM (#20621337)
The behavior is even worse if you leave swap enabled, as I already detailed in my post...
Re:The Linux kernel's inability to gracefully handle low mem... by Aaargh20318 on Tuesday August 06, 2019 @01:14AM (#20622344)
Isn’t iOS basically a flavor of macOS that runs without swap ?
Re:The Linux kernel's inability to gracefully handle low mem... by earenndil on Tuesday August 06, 2019 @01:35AM (#20622444)
Yes, but it's also a heavily integrated environment that aggressively quits background programs on memory pressure.
Re:The Linux kernel's inability to gracefully handle low mem... by Aaargh20318 on Tuesday August 06, 2019 @04:17AM (#20623257)
But isn't that exactly what the linked article advocates Linux should also do ?

Before quitting background applications it first sends them a request to free memory, in a well-behaved iOS program you use this to clean up your caches and ensure your don't use more RAM than you absolutely need. You should also suspend your state to disk when your app is backgrounded so you can just continue where you left off if your app is killed.

Many macOS apps also do this, you can forcefully restart a Mac and after a reboot it'll restore your session to pretty much the exact state you left it in, including any open 'unsaved' files.

Linux could implement a similar mechanism to signal apps to clean themselves up and maybe a 'save your state, you're about to get killed' signal.

Re:The Linux kernel's inability to gracefully handle low mem... by ken on Tuesday August 06, 2019 @12:36PM (#20626788)
> Linux could implement a similar mechanism to signal apps to clean themselves up and maybe a 'save your state, you're about to get killed' signal.

Isn't that pretty much what "memory.pressure_level" [1] is?

[1]: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.tx...

Re:The Linux kernel's inability to gracefully handle low mem... by codedokode on Tuesday August 06, 2019 @02:28AM (#20622707)
That is the only working solution with HTML and JS based applications or apps using GC. Applications should save the state and quit (or be ready to quit) instead of using memory in backround mode.
Re:The Linux kernel's inability to gracefully handle low mem... by floatingatoll on Monday August 05, 2019 @09:21PM (#20621140)
Does the time until oomkiller change dramatically on spinning metal versus SATA SSD versus NVMe, in a swap-off scenario?