The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).
A better fix would be to purge old unused kernels automatically but as far as I understand there were some difficult edge cases around that.
Sure, I'll just use 1/6th of SSD to store 60 megabytes.
$ du -hs /boot/
56M /boot/
If 512M is not enough space for /boot you're doing something wrong.I don't know what planet you're living on but it's certainly not this one. Between a Ubuntu desktop, a laptop and personal server with multiple Ubuntu VM's on it, all of which are kept up rigorously to date, I fix this problem at least three times a year, every year.
The command line process to fix it[1] is a multi-stage mess of dense bash-foo that comes with a 140 word, two paragraph explanation so that /ubuntu veterans/ can figure out what is going on without resorting to scouring the man page for flags. The friendly GUI process to fix it relies on a third party tool that is no longer maintained[2].
It is not possible to explain to non-technical users what is happening here, which means the only thing they can do when they see this is call their technical friend and cry for help. This is exactly the kind of user experience that makes people think Linux is not ready for widespread desktop use.
This is definitely something the OS should take care of itself. I'm ignorant of the challenges that caused it to be this way in the first place, but in my ignorance I would advocate that:
a) the partition be made larger by default b) the OS auto-purge any kernel package more than three revisions old
[1] https://askubuntu.com/questions/89710/how-do-i-free-up-more-... [2] https://launchpad.net/ubuntu-tweak/
(warning, rant inside)
IOW, it does automatically remove old kernels, it just keeps the last 2-3.
So, yes, run "apt-get autoremove", that's it.
I love having a terminal with bash and use it constantly, but I don't think it should be needed for the system to just go on working.
Which is not to say you're lying, I'm just sort of flabbergasted that this is an issue for so many people. Do you run autoremove much? Maybe that would solve it for you?
sudo bash -c "apt auto-remove --purge; apt update; apt upgrade" is what I usually run.
Prefer they focus engineering cycles on actual engineering problems.
And I believe my boot partition is only 256MB, and I didn't set it to that. That was a system default.
Super common in enterprise deployments. I ran into this a bunch on my $EMPLOYER-issued workstation.
I understand that it may be wise to keep the old kernel around so the system can be booted in case there is a hardware incompatibility or breakage in the new release, but that justifies only one additional kernel. Ubuntu keeps those kernels sitting there until you `apt-get autoremove`, and that means that unless you're running that command routinely, the boot partition is going to fill up at some point, no matter how big you make it.
This is especially a problem for people who use the unattended-upgrades package. I've autoremoved and had it clean up almost a gig of old kernel images before.
The update process generates on the order of 100mb/month.
> apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
linux-headers-3.19.0-79 linux-headers-3.19.0-79-generic
linux-image-3.19.0-78-generic linux-image-3.19.0-79-generic
linux-image-extra-3.19.0-78-generic linux-image-extra-3.19.0-79-generic
0 upgraded, 0 newly installed, 24 to remove and 39 not upgraded.
After this operation, 1,732 MB disk space will be freed.
Do you want to continue? [Y/n]Currently, Ubuntu installs a new kernel and update-notifier tells the user a reboot is needed. The autoremove notification only appears when using the terminal which explains why users are running into this issue. Also, update-notifier informs the user another reboot is needed after autoremove is run.
To avoid this mess I’ve commented out the lines of /etc/apt/apt.conf.d/99update-notifier and wrote my own updater using bash and zenity and incorporated needsrestart. It’s not pretty but it works.
For example I'm using LVM with my installation on my Ubuntu laptop and after updating the kernel and running "apt autoremove" it removed the LVM package leaving me scratching my head shortly on reboot as to why it wouldn't find my root filesystem (frankly i have no idea how it became "unneeded").
A more sensible approach is how Red Hat do it with YUM/DNF, that is, to allow a certain number of the same packages to be installed, "installonly_limit" in yum.conf. Doing this means that when a new kernel gets installed the oldest is removed to keep the the system at the limit specified.
On my RHEL/CentOS machines I tend to narrowly provision /boot to around 250-500MB. set "installonly_limit" to 2 and the system will keep the most recent kernel and one back. it works for me.
Of course it doesn't! Why would you assume such a silly thing? /s https://askubuntu.com/questions/563483/why-doesnt-apt-get-au...
Anyway sometimes one wants to keep old kernels. I have an old laptop that runs OK with a 3.something kernel and has wierd video sync problems with any newer ones. Ubuntu 16.04 keeps running with that old kernel so I keep booting from that, maybe once or twice per year.
However the proper solution would be pinning a package and autoremoving the others.
But Muh Freedoms! I hate to be subject to one man's opinion of things /s
People who know they have broken kernels don't keep upgrading them, they stop and fix them.
People who don't know they have broken kernels also don't know they can boot with an older kernel, so they get nothing from the "backup".
We want to leave some time for people to realize their kernel is broken, so keeping three is probably just fine. Honestly, it would probably be adequate to just bump the oldest one off the queue whenever a newer one is requested. If you've got a tiny boot partition, maybe that means only two revisions. If you've got a huge boot partition it could be 20.
But just keeping them all and making people manually uninstall them gains you nothing, it's user-hostile for no reason.
Once you reboot/kexec into the N+1 kernel, it'll let you remove the N (now N-1) kernel, bringing you down to one. But at that point you've proven the new kernel works—at least well enough to get to a shell you can run apt autoremove from.
This is why autoremove isn't so auto: if it happened automatically after reboot, it might be running on a now-wedged system (e.g. one that can't bring up the display manager), removing the last-known-good kernel and leaving you with only the broken one.
I think the right middle-ground solution would just be for installing kernel updates to touch a file, and for Desktop Environments to notice that file and trigger a dialog prompt of "you've just rebooted into a new kernel. Everything good?"—where answering "yes" runs apt autoremove. On a wedged system, you can't answer the prompt, so the system won't drop the old kernel. (In other words, just copy the "your display settings were changed. Can you read this?" prompt. It's a great design!)
That way, you have kernel N running, first autoremove wipes kernels N-1 and older, then it installs kernel N+1, so that when you reboot into N+1, you'll always have known-good kernel N if it doesn't work.
It's a very similar solution to how a good programmer solves an off-by-one error, doing a shift/rotate shuffle on a for/while loop.
I agree that if the user manually runs an apt [dist-]upgrade—or really any manual apt command—that that's a good time to do apt maintenance work. (Homebrew does maintenance work whenever you invoke it and there haven't been any complaints so far.) But kernels usually get installed automatically, so it can't just run then.
Now, if there was a specific concept of a "last-known good kernel" (imagine, say, the grub package generating+installing a virtual package when you run grub-install, that depends on whatever kernel you specified as your recovery kernel, ensuring it remains around), then your approach could work—you'd always have two kernels, the LKG for a recovery boot, and the newest for a regular boot.
I'm running Ubuntu 16.10 currently. A kernel upgrade hosed my setup yesterday, and having an older kernel available saved my butt. I was able to do another `apt-get update` and things eventually worked with the latest kernel.
What? This is ridiculous and unacceptable. I don't use Ubuntu anymore, can someone tell me what is filling up the boot partition?
I'm currently on ArchLinux and mine is 200MB and it's 14% full! I can't fathom what could occupy so much space.
It can be manually fixed by removing older kernels ("sudo apt purge ...").
Perhaps I'm mistaken but i thought a fix was in place for this, maybe it was something third-party but apt definitely offered to remove unused kernel package for me recently.
http://manpages.ubuntu.com/manpages/xenial/man1/purge-old-ke...
Maybe add some stats to know which kernels have booted successfully, so one knows which old ones can be safely deleted, and keep the last 1 or 2 good ones (not always the latest!).
The program 'purge-old-kernels' is currently not installed.
You can install it by typing: apt install byobu
"byobu"? packages.debian.org to the rescue...https://packages.debian.org/jessie/byobu "Using Byobu, you can quickly create and move between different windows over a single SSH connection or TTY terminal, split each of those windows into multiple panes, monitor dozens of important statistics about your system, detach and reattach to sessions later while your programs continue to run in the background."
Uhm... what?
It's completely wrong that purge-old-kernels is in Byobu, rather than in the kernel or directly handled by dpkg/apt. Thanks for all the feedback here -- we'll get that cleaned up in 17.10!
For `apt-get install purge-old-kernels` I get
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
And for `sudo apt-get install purge-old-kernels`, I get Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package purge-old-kernels
But I can reproduce your error by `purge-old-kernels` alone. That is indeed strange.EDIT: I straced bash by `strace -o bashlog -f -s 10000 bash` and found the culprit to be /usr/lib/command-not-found. Indeed, if you run `/usr/lib/command-not-found -- purge-old-kernels` directly, you get that same message about byobu.
EDIT2: I assumed this was some kind of bug with the database, but now I actually tried installing byobu and it does make purge-old-kernels available.
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
^^^^^^^^^^^^^
In your case, you didn't prepend `sudo`.Regarding the second, which is instead the core of the problem, see the man page (http://manpages.ubuntu.com/manpages/xenial/man1/purge-old-ke...):
Provided by: byobu_5.106-0ubuntu1_all bug
Therefore the correct (and full) command is: sudo apt-get install byobu
As pointed out in another comment, such package is arguably a poor placement for this type of utility.You don't ned to use packages.debian.org to look up what packages do.
> apt show byobu .. Description: text window manager, shell multiplexer, integrated DevOps environment Byobu is Ubuntu's powerful text-based window manager, shell multiplexer, and integrated DevOps environment.
* http://packages.ubuntu.com/yakkety/all/byobu/filelist
(-:
But yes, you're very right. It needs to be moved out.
Only Ubuntu does that, other linux distributions don't have this problem.
I recently installed Ubuntu on some old computers for my relatives and they really like it. If they keep updating and after a couple of months their system fails to work that will be a disaster for any good will they will have developed for Ubuntu.
This needs to be fixed NOW! How can Ubuntu even pretend to be a viable desktop operating systems if normal updating renders the system unusable?!
echo $(dpkg --list | grep linux-image | awk '{ print $2 }' | sort | sed -n '/'`uname -r`'/q;p') $(dpkg --list | grep linux-headers | awk '{ print $2 }' | sort -n | sed -n '/'"$(uname -r | sed "s/\([0-9.-]*\)-\([^0-9]\+\)/\1/")"'/q;p') | xargs sudo apt-get -y purge
sudo purge-old-kernels --keep 3 -qy
TILI eventually set up a cron script to regularly delete everything except the last 3 kernels. I think this should really be the default behavior. "Save all the old kernels until you run out of space and everything crashes" doesn't sound like a very sane default.
$ apt-get dist-upgrade
$ apt-get autoremove
dist-upgrade installs new kernels, and autoremove will automatically remove old kernels (and keep the last 2 most recent ones.)As for the original issue of not cleaning up the kernels, this is fixed in xenial/16.04 but not in trusty/14.04, see bug report here: https://bugs.launchpad.net/ubuntu/+source/update-manager/+bu...
dist-upgrade is NOT (in most cases) meant to upgrade to a newer distribution (eg. Ubuntu 14.04 to 16.04). dist-upgrade is just like upgrade except it also installs additional packages if necessary (eg. linux-kernel-4.1 AND linux-kernel-4.2).
Most people should always run "apt-get dist-upgrade" and never "apt-get upgrade" in order to simply keep their packages up-to-date.
An actual distribution upgrade is triggered by a different command. On Ubuntu: do-release-upgrade
This is a feature (kernel update) that is used by everyone, including newbs. They don't have to understand anything to enjoy the update, and they shouldn't have to understand anything to avoid the space filling issue.
Lots of ways to fix it, I don't claim to know which is best:
* Always leave N% /boot available, and delete or move old kernels to satisfy that.
* Move old kernels to /old_kernels, outside of /boot. Driven either by satisfying N% space, or no more than K number of kernels kept in /boot.
* Opt in, opt out, ask this on installation and for non-server installs, default to "never have to think about this again."
* Easily configurable in the "whatever that box was called when I last used Ubuntu years ago" box.
really, this is what your package manager should do automatically before/after installing new kernels
for your reference: http://askubuntu.com/questions/2793/how-do-i-remove-old-kern...
Just run "apt-get autoremove".
The relevant bug report is here: https://bugs.launchpad.net/ubuntu/+source/update-manager/+bu...
Side note, give up on a separate /boot in future, not needed 99.9% of the time.
You could probably fix this on your system without installing from scratch, but it would take some careful planning (mostly wrt backups!). You'd need to boot from a recovery cd or such, copy /boot into /, edit /etc/fstab accordingly, and then follow the last few steps in the bug report above (from the chrooting onwards). I'd probably test this in a VM first.
I manually removed a bunch of old initrd images the other day.
http://askubuntu.com/questions/616512/purging-old-kernels-fa...
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717584
They've got a bug reported upstream. We need this fixed in Ubuntu
https://bugs.launchpad.net/ubuntu/+source/dkms/+bug/1515513
You can add yourself to the list of people affected to increase the Bug Heat and get this issue fixed.
$ mountpoint /boot
/boot is not a mountpoint
I don't know if this is the default, but my KUbuntu machines have been fine for many years without a separate /boot.In theory, GRUB can load kernels off of a LUKS-encrypted partition, but in practice I've never managed to set that up without having two passphrase prompts, one from GRUB and one for mounting the root filesystem under Linux.
If people are encountering it on this wide a scale, it must be a truly severe problem (that's also probably causing many people who haven't learned about apt-get to give up on software updates entirely!).
https://gist.github.com/vzaliva/bee68ed709ddd2724791a56714ec...
This more than anything. I believe it is this issue (at least for me): https://bugs.launchpad.net/ubuntu/+source/unattended-upgrade...
- HEADLINE: Please, please, please fix space issues with /boot.
- DESCRIPTION:
I'm constantly running out of space in /boot, due to kernel updates. It drives me so incredibly batty. If I had to guess, this is due to poor defaults in the installer for folks that opt to encrypt their whole disk. Even still, this system was setup back on 14.04 (don't think it started on 12.04), and I have no intention of reinstalling from scratch just to fix it.
Publish something official on how to fix this problem! Make it easy and stress free! Yell at the people who didn't catch this bug before it went out! Sorry, but this is just a really bad problem: it leads to folks like me wasting time, and probably a whole bunch of other folks just not being able to install updates, and no idea why.
- ROLE/AFFILIATION: software developer in the federal government