Yeah, Ubuntu still keeps running out of space in /boot by filling it w old kernels in the default partition profile. A bug that I've been hitting since about 2011.

In spite of their best efforts, it is not the "grandma friendly" quality software it hoped to be.

If we're sharing Ubuntu bugs, how about the one where NFS with Kerberos had been completely broken since several years back. You simply can't mount such a share, and you'll always get an authentication error.

I reported it as soon as I upgraded (to 14-something I think) and the bug was marked as severe. Every time a new release of Ubuntu comes out I check if it had been fixed, and of course it hasn't.

FWIW, we've been able to use NFS with sec=krb5 on both 14.04 and 16.04.

My M.O. is:

  # apt-get update
  # aptitude full-upgrade
  # apt-get autoremove

Because as far as I can tell, apt-get sometimes fails to actually update things, whereas aptitude can't autoremove. I'm sure there's a better way.

This is my major gripe with the dpkg ecosystem. There are (AFAICT) multiple different tools to build dpgks and multiple different tools for updating your system, and none of them are adequately documented nor support all use cases.

The Fedora ecosystem is better in this regard. Want to build an rpm? rpmbuild is the only answer. Want to update the system? Use dnf. (Sadly, there's now also PackageKit, and it's not quite isomorphic to dnf.)

AFAIK, the only supported way to build a deb is using dpkg-buildpackage, and all the others just wrap it (unless we're talking about alien, but that's not really "supported" going any direction).

Updating your system is all the same backend, and the Debian-alikes try to tell you to only use one tool for cmdline (apt, now, which just takes the same args as all the various apt-FOO commands did, all of which still work), and one for the GUI (Synaptic, or wrappers around it, unless Ubuntu Software Center has mutated a lot since I last looked).

IIRC they made a big deal about not installing aptitude (a popular alternate text UI for apt-based systems) by default on Ubuntu _because_ they wanted to standardize and fix any deficits, rather than working around them.

> they wanted to standardize and fix any deficits, rather than working around them.

That would work great if would they would take bugs reports seriously. In practice, with Canonical I file a bug and ninety days later get an automated message that the bug was closed due to inactivity.

That's an unfortunate pendulum swing cleaning up from their problem of yester-year - namely, bugs sitting around _forever_ without any comment.

I noticed they changed that policy when I started getting comments on years-old bugs of closed-obsolete. If they're now 90d expiring bugs, I'd _guess_ there's a bit that can be set for your account of whether you're a customer paying for support, and that their support staff have a queue filter that prioritizes those, and the rest get expired if nobody comments.

> apt-get autoremove

You should be doing:

    apt-get autoremove --purge

Otherwise you'll be leaving a lot of clutter around (obsolete configuration files).

apt-get update won't do anything beyond download the latest view of the repository.

you want:

apt-get update && apt-get dist-upgrade && apt-get autoremove

Aptitudes full-upgrade is synonymous with dist-upgrade.

Others wrote "apt-get dist-upgrade" in sibling comments, which should do the trick. If you want to build packages for multiple managers, fpm is a nice front-end for some of them: https://github.com/jordansissel/fpm

I use FPM all the time, but it is not a 'nice' way to package things, it's an 'easy' way to package things ;)

> apt-get autoremove

This is not going to autoremove kernels unless you modify the default preferences in /etc/apt/. For added benefit you should probably add --purge to your autoremove command.

apt-get dist-upgrade is what you nerd to use to upgrade all packages... Don't blame apt-get for your lack of knowledge on how it works

I lost my encrypted backups thanks to the Ubuntu installer. It turns out that when you set up an encrypted volume, the installer overwrites the LUKS metadata block on all connected volumes. I happened to have my encrypted backup volume connected.

As great as Ubuntu is, I've always thought that a project of their magnitude and their resources (which I grant is not a lot) should focus on QA, specifically with these sorts of issues.

Yep, I had to fix a 14.04 server like that last week. /boot filled with old kernels and apt was falling. I had to manually fix that.

I also don't have swap on my laptop (16 GB RAM and SSD) and I rarely hit 10 GB of used space. With no swap I mean no swap partition. I'm getting those out of memory errors in Opera tabs since the new kernel. Luckily my primary browser (FF) is fine. I guess it's trying to swap Opera out even if there is no swap. I'll reboot with the older kernel and see what happens.

Likely, opera asks for a disk-based anonymous memory mapping - this goes on the swapfile if you have one, and fails if you don't.

The /boot bug is unreal, I just ran into that a few months ago.

It's super frustrating. There's no reason I need twenty kernels on my machine :-p

Ubuntu actually only keeps the current and the previous kernel as necessary and give the user the option to purge all the old kernels. These old kernels will be marked as autoremovable and by setting a flag in unattended upgrades config you can make it autoremove those. On desktops this happens automatically.

On RHEL, it happens on servers, too - anything is better than your /boot partition filling up on its own!

It shouldn't, the default is to keep 6 recent kernel packages.

Honestly, the amount of regressions, packaging problems, hacky fixes, non-stardard configuration and poorly set defaults in Ubuntu is disgusting, look at how bad NetworkManager was / is on it, it's not that bad on Fedora (not that I like it that much) and the whole debarcle with them doing Unitiy on top of Gnome rather than contributing to the community projects, or recently when they released an update that broke all USB 2 ports and if you got that update before they pulled ita and rebooted - you had no keyboard or mouse to work with, their own custom (IMO very gross) init system of Upstart when they could have contributed to System which they're now going to move to anyway, their dodgy partnerships with companies like Amazon where they're putting apps / links on the desktop menus to Amazon (who does very little for the open source community also), it's just a joke and I really wish people would just move on to better distros or at least use pure Debian if they can handle the lack of packages. So often at work when devs complain their DNS isn't working or they can't connect to the VPN or they updated and now their machine doesn't boot - our first questions - 'are you running Ubuntu?' it's like the Windows of the linux world I swear.

I think that Gnome3 has a more detrimental effect on community projects than Unity. Why? Because of the client side controls they are pushing, rendering Gnome apps for everything non-Gnome3 either useless or ugly. And all of that for a few pixels of screen space, something that has been solved more elegantly by putting menus in the top panel.

And talking about breakage: I can't count how many times some important Gnome3 plugin broke after a minor Gnome3 update from the distribution. Or how many times I wasn't able to install a plugin that installed fine just a month ago on the same Gnome3 version. Why? Because you have to download them from some obnoxious website that apparently has no archive and no way to properly match the right plugin version to a given Gnome3 version.

Cinnamon copied the same madness. But you have to install less addons to make it at least usable. KDE is somehow still stuck in the previous decade when it comes to desktop design.

If the Gnome3 project with their questionable design choices and holier-than-thou attitude is the future of desktop Linux, then I have to say that the situation is much worse than it was ten years ago.

BTW, Ubuntu doesn't come with Amazon addons installed anymore. And even when it did, you could entirely remove it and look at the source. I think Canonical realised that they cannot pull something like that anymore.

> rendering Gnome apps for everything non-Gnome3 either useless or ugly.

I use i3 instead of Gnome 3 and the Gnome 3 apps with CSD fit in beautifully. There used to be issues where there were gaps and such, but those are resolved by now.

> something that has been solved more elegantly by putting menus in the top panel

Arguable and probably comes down to personal preferences. For some apps (like the screenshot tool) it fits really well.

andrewshadura is right. Here's just one of your items that is bunkum, for an example.

> their own custom [...] init system of Upstart when they could have contributed to System

upstart was developed several years, just under half a decade, before systemd even existed; it was used by Fedora for three years.

It's true though that Canonical consistently "does their own thing" and then fails to establish it as a standard for whatever reason (technical, political, CLA...). Anyone who adopted the new thing then has to migrate.

Eucalyptus vs OpenStack (ever heard of Eucalyptus?)

Bazaar vs Git (Git won)

LaunchPad vs Github (LaunchPad is alive and well, but Github won the popularity contest)

Wayland vs Mir (Wayland won)

upstart vs systemd (systemd won)

Let's see how these work out:

Juju vs Kubernetes/OpenShift (I'm aware that you can run Kubernetes on Juju, probably won't save them though)

Snap vs Flatpak (vs AppImage?)

It's sad, really. I like Ubuntu and Canonical and strongly believe in competition, but it'd really help if they didn't consistently back the wrong projects.

In a lot of those cases (Mir being the obvious exception), the Ubuntu backed solution predated the alternative.

I can't help wonder if this will keep happening as their competition waits to see everything new Canonical chooses to back, then works to thwart it :)

Yeah they invent a new wheel (often poorly) then they fail to test and fix it when it breaks.

I guess they need to buy more project managers...

And if Canonical hadn't required contributors to sign a CLA... https://news.ycombinator.com/item?id=7640930

a short point of order: Upstart existed and shipped long before Systemd existed, Infact Red Hat Enterprise Linux 6 shipped with Upstart. It was the shortcomings in Upstart that led to the development of Systemd. If you're using RHEL 6 you're using Upstart.

So much written, so little sense in that.

Have you actually looked at Ubuntu, have you tried to use it? I'm not even going to go through your assertions one by one to debunk them, as most of them are wrong and unsupported by facts.

Cheers.

he got some details wrong but the frustration is real, especially if you have anything slightly off standard, like a multi monitor setup or, god forbids, a touch screen.

I have a touchscreen. Works great nowadays.

even pressure?

That I wouldn't know. It's not capacitive.

Been using Ubuntu for years in multiple production environments and I agree with him.

Much fewer issues with Red Hat.

My mother, who is literally a grandmother, ran into the /boot old kernel's issue twice already. Very annoying.

I used to use this to clean boot. dpkg -l 'linux-' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]\)./\1/;/[0-9]/!d' | xargs -p sudo apt-get -y purge

What does it do?

I tried it on Lubuntu 16.04; it required modification:

   # mystery one-liner from https://news.ycombinator.com/item?id=13513171
   # dpkg -l 'linux-' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]\)./\1/;/[0-9]/!d' | xargs -p sudo apt-get -y purge
   # my corrected/improved version:
   noncurrent_kernel_pkgs() { dpkg -l 'linux-*[0-9]*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.\)-\([^0-9]\+\)/\1/")"'/d;s/^ii *\([^ ][^ ]*\)[^ ]*.*/\1/' ; }
   # corrections/(possible-)improvements:
   # - dpkg -l 'linux-' # returns nothing
   # - dpkg -l 'linux-*[0-9]*' # moves 'pkgnm must contain a number' rule (last sed cmd in orig: '/[0-9]/!d') forward
   # - pkgnm isolation/extraction was broken (did dpkg output change?)

output:

  kg@kg-VirtualBox:~$ noncurrent_kernel_pkgs
  linux-headers-4.4.0-43
  linux-headers-4.4.0-43-generic
  linux-headers-4.4.0-45
  linux-headers-4.4.0-45-generic
  linux-headers-4.4.0-53
  linux-headers-4.4.0-53-generic
  linux-headers-4.4.0-57
  linux-headers-4.4.0-57-generic
  linux-image-4.4.0-43-generic
  linux-image-4.4.0-45-generic
  linux-image-4.4.0-53-generic
  linux-image-4.4.0-57-generic
  linux-image-extra-4.4.0-43-generic
  linux-image-extra-4.4.0-45-generic
  linux-image-extra-4.4.0-53-generic
  linux-image-extra-4.4.0-57-generic
  kg@kg-VirtualBox:~$ uname -r
  4.4.0-59-generic
  kg@kg-VirtualBox:~$

Sorry about that, the '*' got removed from a non marked up paste. You are 100% correct. Just a little script to remove all but the current unused kernels that I use on 14.04. Yep, just take the xargs off to see what it will remove (which would always be good advice when pasting a random script LOL ).

Take off the xargs part of the command and run it to see. :-D

Just looking at it though, it looks like it'll purge kernel packages that are the currently booted one. I don't have a system handy to check though so take that with a grain of salt.

Just for posterity that should have been "are not the currently booted one".

I downvoted you for "grandma friendly." My grandma remembers when LaTeX was "new" and was one of the first people to professionally use computer-based typesetting. What exactly does "grandma friendly" look like to her?

I utterly agree that it's patronising to use a generic grandma as an example of a naïve user, and we should stop doing it.

However, I suggest that your grandma's long experience with computers probably means that she in would be especially displeased with the crummy state of the Linux desktop in 2017!

I just wanted to say I think it's sad you're getting downvoted. My grandma was also relatively computer literate. If we ever hope to reach equality in our line of work we need to think more carefully about the personas[0] we use, not because our grandmas care but because of how such descriptions affects the people who might one day fall into that user category.

[0] https://www.usability.gov/how-to-and-tools/methods/personas....

See also https://news.ycombinator.com/item?id=11667485 .

Fair enough.

I just use the term because I'm "family tech support" for multiple grandmas. I'll avoid that phrasing in the future.

Thank you! :)

You can run `apt-get autoremove` and it will purge old kernels/headers for you (amongst any other no-longer-used dependencies).

Oh, I know it's easy to fix, I just single that bug out as "I have done nothing with this box but install updates and surf the web. I am Grandma. It breaks".

If you're Grandma you don't have /boot on a separate partition, you have all / on a big 120GB at least, so if you run out of space you simply know you've run out of space.

Not perfect but not a problem for 90% of people I guess...

Ubuntu installer has separate /boot partition by default. If you don't do anything special during installation you end up with it.

No you don't. Ubuntu's default partition map is one big / and one efi(if necessary).

I literally just did a BIOS install and it has a separate /boot.

I assume it changed recently, I've seen fresh 2014 installs with tiny 255MB boot partitions.

This was ubuntu's recommended/default partition plan. I did not customize it. Now, it's possible this is a plan they no longer recommend and have changed since I first installed Ubuntu, but the point stands:

I installed Ubuntu onto a machine, chose the recommended default options, and it set up a /boot partition that was quite small, and its normal update cycle fills it up and fails to fix the problem automatically or prompt the user with information to fix the problem.

My old mother (born 1936) is using ubuntu since 2013. She can read her mail, answer or forward, view powertpoint, video and pictures, web surf, print stuf, write letters (its her most complex task). I use ssh and x11vnc for remote administration. It is the most simple setup I can imagine.

These are autoremoved nowadays.

> In spite of their best efforts, it is not the "grandma friendly" quality software it hoped to be.

Grandma didn't customise her installation to put /boot in its own, small partition.

FWIW, you can run "apt-get --purge autoremove" to remove old kernels since at least 16.04 and maybe 14.04 (I don't recall). Not fully automatic, but at least it is one command. For fully automatic, you can install and configure unattended-upgrades and configure it to autoremove automatically.

I don't remember when I first installed mine... 12.04 maybe? But this was the default/recommended partition plan.

IMHO if you can't not use swap without running out of memory in normal workloads, you either need more RAM or to work on reducing your memory consumption; swapping is for temporary overloads or gracefully recovering from one, not a substitute for RAM.

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result.

That's a surprising behaviour change (from the perspective of "don't break userspace"), but looks more like a "fix of previously broken behaviour" as in 0 not being able to disable swap entirely; from that perspective, using 1 makes sense.

I think that the new behavior at zero (documented anywhere?) is a discontinuity. There should be a separate setting for mlocking code pages.

Swap is useful even if you have plenty of RAM. It's better to swap out a cold file-backed page and use that page for cache instead than to pin that cold page in memory for no reason.

Gah. I meant non-file-backed

That's what ZRAM is for.

It's also what swap is for.

Funny, but Linux is the only unix-like OS I've ever run where using swap means you have a problem. Other OSes have much more intelligent memory management.

For me swap is an error. I would rather a server crash then go into swap and run slow.

If your servers are at 75% CPU and start swapping you are screwed. Time to start adding servers. In this day and age you are wasting a lot of money to be sitting around waiting for swap.

zswap makes swap useful on the server as it results in gradual slowdown when the system has used all the ram. And if the system runs at 75% of CPU, then there is room for such slowdown.

I hadn't heard of zswap [0]:

"Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them into a dynamically allocated RAM-based memory pool. [...] Zswap is a new feature as of v3.11 [...] zswap is a work in progress and should be considered experimental."

The return of RAM Doubler!

The naming is initially confusing if you're already familiar with ZFS, especially:

"Zswap makes use of zpool for the managing the compressed memory pool"

[0] https://www.kernel.org/doc/Documentation/vm/zswap.txt

It's actually quite useful on many consumer devices, especially those with multiple cores. Most of the time I'm happy I can trade a few CPU cycles for better I/O performance.

yes, we run ZRAM of every single server we have, its fantastic and we don't have swap anywhere. It's worth noting though that vm.swappiness is set to 0 then ZRAM wont be used, it must be set to 1 or higher unless you're using a really old kernel (pre 3.5).

I have found that zswap behaves much better than zram when the size of the used ram approaches 150% of RAM. In theory zram compresses better as zswap has very simple mapping for the location of its cache of compressed pages leading to space underutilization. But, as it has real backing store, it can evict unused pages allowing bigger chunk of uncompressed memory for running applications.

zram[0] is not the same thing as zswap[1].

0. https://en.wikipedia.org/wiki/Zram

1. https://en.wikipedia.org/wiki/Zswap

It's not the case on Linux either.

Linux intelligently swaps out ununsed pages even if there's enough physical RAM (that is more useful as page cache).

I have FreeBSD systems that use gigs of swap with gigs of memory free. If you have a Linux system with that much swap usage it's usually about to crash (as it struggling under memory pressure)

People should stop monitoring swap usage and start monitoring the amount of page in / page out IO.

Not true. I have plenty of Linux systems that use gigabytes of swap with 100+GB of free RAM.

Monitoring swap usage doesn't make sense on either FreeBSD or Linux.

heh, try using Windows when it goes into swap, Windows both desktop and server additions have the worst memory management I've ever seen and having no ZRAM and a pig of a filesystem like NTFS really doesn't help either.

Going into swap in Linux isn't a problem, it's a symtom of a misconfigured / provisioned server or an unhanled application memory leak, other than that the other only things that should cause it is incorrectly configured use of shared / reserved memory.

Disk based swap should pretty much always be disabled these days and ZRAM used to incresed efficiency.

> having no ZRAM

Windows 10 has had memory compression [1] for quite a while now. On the PC I am on at the moment with 100+ Chrome tabs open, I have 1.3GB compressed out of a total of 16GB.

[1] http://www.thewindowsclub.com/memory-compression-in-windows-...

Works very well on Windows Server 2012 and saves massive amounts of money with some workloads.

Incidentally, at least on Windows, you can save a ton of harddrive space by changing your swap settings - the default is crazy high, especially if you have lots of RAM.

My laptop has 8 gigabytes of ram. I can fit all of my personal files in that at obce, I don't see why that's not enough for everything.

Space used on disk is less than when the data is in RAM. It's not a binary copy loaded. It's binary data loaded then inflated into app-object memory. So much overhead

I’ve run with 4 GB of memory and no swap for a long time. It’s not hard.

Swap makes a lot of sense for a consumer device, where I'm opening a big spreadsheet, then putting that in background to edit this video.

It makes less sense for server workloads, where you usually don't have huge chunks of inactive memory.

I always run with no swap in AWS, especially with the newer EBS-only families.

EBS is a problem because in reality you would be swapping to a network volume. But on a standard VPS server, a bit of swap can really save you some problems. Swap was invented at a time when *NIX was strictly used in a multi-user environment, not far removed from today's webservers but very different from today's consumer devices. It still serves a purpose in that role.

> [...] a bit of swap can really save you some problems.

And cause a bunch more. Have you ever had to debug a race that is only triggered when a process hits swap. Not fun at all.

Yes I did, and the swappy machine started responding just a tad too slowly to other nodes' pings, engaging into a loop of join/parting the cluster, causing replication to kick in all the time and a whole "highly available" critical cluster of 9 nodes was unable to handle traffic (too busy replicating) causing millions in losses. So yeah, don't use swap in servers.

Ironically, if the node had crashed for OoM, the cluster would have survived just fine.

>"fix of previously broken behaviour" as in 0 not being able to disable swap entirely

But you can disable swap entirely by not having a swap partition/file, so I don't think that's a huge issue.

It was indeed a massive, not well discussed change and the kernel maintainers should have vocalised it better (not that I think the Ubuntu maintainers wouldn't understand it properly anyway TBQH) but yes, the behaviour was previously logically incorrect and had to be fixed at some point.

Exactly.

I have stopped using it long time ago, after they introduced the following bug:

https://bugs.launchpad.net/ubuntu/+source/netcat-openbsd/+bu...

Crikey - they did what?!

I recall a fascinating article / thread years ago about swappiness=0 not actually meaning what people thought it meant, and explaining why it was (generally) a bad idea. A low number, yes, but not 0.

Doing some research now, and found a blog written by someone bitten by this change back in 2014-04 -- referring to the commit that introduced this in 3.5-rc1. Not clear why the change was made, though, as a) it breaks lots of people's understanding / expectation, and b) if you want to never swap out then just forego the creation of a swap file / partition.

[1] https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...

Exactly, people in this thread missing the point of the use of RAM. A perfect operating system always going to use as much RAM is available for caching and there are long running processes that can be swapped out to free RAM for more important use cases. Memory management and paging/swapping is an interviewing topic for SREs/systems engineers, and this is not an accident. Oversimplifying the problem does not help.

It's surprising that many professionals have broken mental models about how an operating system supposed to work. I think they have something like DOS in mind. I was just told last week about how caching static (!) files from web server in redis increase performance in their infrastructure. I was baffled.

Yeah it wasn't well announced when the change was made, esentially if you find any linud machines running any kernel dating way back to 3.5 make sure you change vm.swappiness=0 to vm.swappiness=1

What are the many missing packages on Debian? From my experience Centos looks like a bad joke.

Most SELinux policies, Corosync and Pacemaker and related HA / clustering tools, and a pile of other things and they settled for a really awful stable kernel major in Jessie / 8.x.

It really went downhill with, the release of 8.0, the only improvement we really saw was a slightly newer kernel (but still old) and SystemD (although not consistantly implemented), The Debian community has also fallen through the floor, lots of smart-ass bickering and slack package maintainership and generally unhelpful, rude contributors / owners. By contrast RHEL/CentOS sped up their release cycles after re-in-housing Fedora as and taking a lot of it's rolling updates into stable a lot quicker and the community has been fantastic, Redhat have really stepped up and modernised their game a lot over the past two or there years.

I happen to run most of the latest of things on CentOS (server-side only, though). My approach has been to package and deliver modern versions of various runtimes (python, ruby, jre) into specific directories for my apps. This way, I am not dependent on the system-provided runtimes which are an older version.

For my work with Belenix (revamp not announced yet), I am working on examples where users no longer depend upon system-supplied package versions, and instead choose their own local instances.

Are you aware of the "software collections"? That may or may not be of use to you.

Why don't you use Software Collections to get the newer versions of the runtimes?

Though I had a commercial contract in place for Software Collection, these were not as up to date with what the communities released (e.g. Ruby). Hence my approach of packaging and testing my own builds of these.

Not sure what you mean, if RedHat/CentOS is good for Amazon to run their 2M linux nodes, it is probably good enough for me to host my services. The stability and production readiness of the platform beats all the other Linux distributions, especially Ubuntu and I would argue Debian too (hello purify complains bug).

As someone who is using all of them, I'm inclined to agree.

CentOS only includes packages that Red Hat builds and supports on RHEL. Extra packages are maintained by the community as part of Fedora EPEL.

and a number of other useful community / product maintained repos such as PostgreSQL's official yum repo, ELRepo, Docker's official repo, Sernet's official Samba repo, Nginx's official repo etc...

What the hell? What happened to "don't break userspace"?

Yeah I know right, Ubuntu breaks userspace all the time, Debian isn't quite as bad but it's much worse than it used to be thanks to so many missing packages, try building a cluster / HA system on it - you actually can't!

RHEL has had a similar behavioral change during lifetime of a release so this particular case is not actually a good example for recommending RHEL over Ubuntu for stability.

https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...

Can you please share your experience a bit more ? How much SELinux is really required ? What about grsecurity.

My use case is custom build Django/Postgres app.

SELinux is an absolute must in todays world, simply put it prevents applications and namespace from doing thing outide of their area of interet, for example a web server should never be allowed to execute files from /tmp so it prevents that, another great example is when you have any form of multi-tennancy - for example you might have several docker or LXC containerised apps running on a host SELinux will / can ensure that even if of those apps is compromised and someone popped a shell in it, they couldn't go beyond anything running in that namespace, i.e. to the host itself or other apps running in containers on the same host.

It prevents a lot of sitations where an app might behave badly, for example perhps you're running a PHP app and a 0-day PHP flaw is discovered, there's a good chance that SELinux restrictions could prevent that exploit from being useful to the attacker by not allowing the app to do unusual things.

We have a rule - if it's not running SELinux in enforicng then it's not going into production.

We use to run GRSec as well, however with modern kernels a lot of the GRSec code base has been merged in so it's often not required.

I agree with you except for the GRSec part - most GRSec features are still missing except for a few low hanging fruit.

grsecurity is enhancing the security of the kernel itself, SELinux is for user space applications.

That's an oversimplification. Grsec protects both kernel and applications, although mostly via separate systems. Selinux mostly protects applications, but can also stop malicious interactions with kernel code (syscall and virtual file access restrictions)

It's not -- I'm getting OOM on some java processes and my swappiness is set to 60.

Java has its own memory limits that it will trigger OOMs at.

I didn't mess with swappiness, but I also didn't have a swap.

Centos/rhel are on a pretty old kernel at this point though?

We run the latest mainline kernel on all our 200-300ish CentOS hosts after it's passed standard testing and is available in Elrepo: http://elrepo.org/tiki/kernel-ml

Back when we ran Debian everywhere I wrote a CI project that would build the latest mainline kernel, applying GRSec and adding some SELinux policies in as best I could (backported from CentOS/Fedora funnily enough) and build them into a Debian package for deployment across our hosts, see: https://github.com/sammcj/kernel-ci

For Centos 7 its 3.10 according to good old http://distrowatch.com/table.php?distribution=centos

and it's whatever is the latest, stable, tested version of the kernel when using Elrepo's Kernel-ml repo: http://elrepo.org/tiki/kernel-ml

Don't use EPEL in production, it is an unstable mess. If you need more packages than are in the core of CentOS, Debian is the best choice.

How is EPEL an unstable mess?