News for hackers. Stuff that matters.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by stsp on Friday March 31, 2017 @09:20AM (#14003402)
+1 -- This is the one and only problem I have to regularly help my non-technical Ubuntu friends (and their friends) with. Every few months they cannot install updates anymore because their /boot fills up and apt fails to install a new kernel package.

The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).

A better fix would be to purge old unused kernels automatically but as far as I understand there were some difficult edge cases around that.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by justinsaccount on Friday March 31, 2017 @10:18AM (#14004007)
> The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).

Sure, I'll just use 1/6th of SSD to store 60 megabytes.

  $ du -hs /boot/
  56M	/boot/
If 512M is not enough space for /boot you're doing something wrong.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by Sir_Substance on Friday March 31, 2017 @12:37PM (#14005499)
>If 512M is not enough space for /boot you're doing something wrong.

I don't know what planet you're living on but it's certainly not this one. Between a Ubuntu desktop, a laptop and personal server with multiple Ubuntu VM's on it, all of which are kept up rigorously to date, I fix this problem at least three times a year, every year.

The command line process to fix it[1] is a multi-stage mess of dense bash-foo that comes with a 140 word, two paragraph explanation so that /ubuntu veterans/ can figure out what is going on without resorting to scouring the man page for flags. The friendly GUI process to fix it relies on a third party tool that is no longer maintained[2].

It is not possible to explain to non-technical users what is happening here, which means the only thing they can do when they see this is call their technical friend and cry for help. This is exactly the kind of user experience that makes people think Linux is not ready for widespread desktop use.

This is definitely something the OS should take care of itself. I'm ignorant of the challenges that caused it to be this way in the first place, but in my ignorance I would advocate that:

a) the partition be made larger by default b) the OS auto-purge any kernel package more than three revisions old

[1] https://askubuntu.com/questions/89710/how-do-i-free-up-more-... [2] https://launchpad.net/ubuntu-tweak/

Re:Ask HN: What do you want to see in Ubuntu 17.10? by lloeki on Friday March 31, 2017 @04:54PM (#14008087)
Here is my old-timey one-liner personal solution[0] for it that has worked flawlessly so far, obscure theoretical edge-cases be damned, because the non-edge case situation is just awfully worse and practically impactful.

(warning, rant inside)

[0] https://gist.github.com/lloeki/520acee8ba3b44c532c7

Re:Ask HN: What do you want to see in Ubuntu 17.10? by kobeya on Friday March 31, 2017 @04:04PM (#14007618)
Um, isn't the fix `sudo apt auto-remove --purge`, which autodetects unused kernels? What am I missing?
Re:Ask HN: What do you want to see in Ubuntu 17.10? by tedivm on Friday March 31, 2017 @06:17PM (#14008842)
If you do not run that command before /boot fills up, and you have a full /boot with a partially installed kernel, then that command fails. So this works fine if you remember to call it regularly, but it does not solve the problem once it occurs.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by kobeya on Friday March 31, 2017 @07:21PM (#14009307)
Interesting. I haven't encountered that edge case. I've many times filled /boot and resolved by doing an auto remove.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by collinmanderson on Saturday April 01, 2017 @10:03AM (#14012150)
It seems silly to me that I need to manage this myself. Why do I need to be worrying about different kernel versions? I just want to make websites.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by Sir_Substance on Friday March 31, 2017 @04:17PM (#14007746)
Re:Ask HN: What do you want to see in Ubuntu 17.10? by alphapapa on Friday March 31, 2017 @05:43PM (#14008584)
Following the chain of links and answers and explanations, we come to the conf file that says in the comments that it commonly results in two (2) kernels being saved, but can sometimes results in three (3) being saved.

IOW, it does automatically remove old kernels, it just keeps the last 2-3.

So, yes, run "apt-get autoremove", that's it.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by ikurei on Tuesday April 04, 2017 @04:56AM (#14030713)
I think it has solved the problem for me, but still is not a good solution for anyone who would answer "What's a terminal?"

I love having a terminal with bash and use it constantly, but I don't think it should be needed for the system to just go on working.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by dsacco on Friday March 31, 2017 @10:50PM (#14010129)
I've been using Ubuntu either part or full time since 2007. I've literally never encountered this.

Which is not to say you're lying, I'm just sort of flabbergasted that this is an issue for so many people. Do you run autoremove much? Maybe that would solve it for you?

Re:Ask HN: What do you want to see in Ubuntu 17.10? by sunstone on Saturday April 01, 2017 @11:21AM (#14012536)
I run ubuntu 16.04 on a laptop, desktop and a TV streamer and I get this all the time. My boot partition on the desktop is 15gig and it gets plugged every now and then.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by bazinga888 on Thursday April 06, 2017 @02:53PM (#14053444)
I've hit this before, but honestly do not think it's a big deal. Sure the installer could default to a larger boot, but it's manually configurable during install. And cleaning it up once in a while is just good sys admin practice.

sudo bash -c "apt auto-remove --purge; apt update; apt upgrade" is what I usually run.

Prefer they focus engineering cycles on actual engineering problems.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by hollander on Friday March 31, 2017 @01:55PM (#14006268)
Sorry, but Ubuntu is doing something wrong here, not me. This should be handled automatically. Ubuntu wants to be the system for everybody, but you can't expect people to open the terminal and fix this manually. Making boot 20GB is ridiculous, but 1GB should be no problem, and for me 2GB would be OK if that means that this problem will disappear forever.

And I believe my boot partition is only 256MB, and I didn't set it to that. That was a system default.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by dustinkirkland on Monday April 03, 2017 @03:36PM (#14026318)
Ubuntu is absolutely doing something wrong here and we'll get that fixed. Thanks!
Re:Ask HN: What do you want to see in Ubuntu 17.10? by cbhl on Friday March 31, 2017 @01:22PM (#14005968)
Yup! That "something wrong" is installing every single kernel update for two, three, four years and not deleting any of the old kernels.

Super common in enterprise deployments. I ran into this a bunch on my $EMPLOYER-issued workstation.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by cookiecaper on Friday March 31, 2017 @02:59PM (#14006995)
The installer should handle this. When you apt-get upgrade anything besides the kernel, does it leave the old version lying around?

I understand that it may be wise to keep the old kernel around so the system can be booted in case there is a hardware incompatibility or breakage in the new release, but that justifies only one additional kernel. Ubuntu keeps those kernels sitting there until you `apt-get autoremove`, and that means that unless you're running that command routinely, the boot partition is going to fill up at some point, no matter how big you make it.

This is especially a problem for people who use the unattended-upgrades package. I've autoremoved and had it clean up almost a gig of old kernel images before.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by lukeschlather on Friday March 31, 2017 @10:40AM (#14004232)
If you're running updates weekly it will fill up on Ubuntu. This is a recent problem, and I've only experienced it on my laptop with full-disk encryption.

The update process generates on the order of 100mb/month.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by jayess on Friday March 31, 2017 @01:27PM (#14006004)
It's not new. It's been happening to me since I started using Ubuntu in the 8.x range.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by fredsted on Friday March 31, 2017 @11:35AM (#14004844)
Doesn't `apt-get autoremove` remove those old kernels? Not that it's a solution; it should of course be done automatically! Here's what I get when using it:

    > apt-get autoremove
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages will be REMOVED:
      linux-headers-3.19.0-79 linux-headers-3.19.0-79-generic
      linux-image-3.19.0-78-generic linux-image-3.19.0-79-generic
      linux-image-extra-3.19.0-78-generic linux-image-extra-3.19.0-79-generic
    0 upgraded, 0 newly installed, 24 to remove and 39 not upgraded.
    After this operation, 1,732 MB disk space will be freed.
    Do you want to continue? [Y/n]
Re:Ask HN: What do you want to see in Ubuntu 17.10? by nominated1 on Friday March 31, 2017 @11:42AM (#14004927)
Whenever a kernel is updated autoremove should be called immediately afterwards. It should be called before the restart now / restart later dialog box of update-notifier appears.

Currently, Ubuntu installs a new kernel and update-notifier tells the user a reboot is needed. The autoremove notification only appears when using the terminal which explains why users are running into this issue. Also, update-notifier informs the user another reboot is needed after autoremove is run.

To avoid this mess I’ve commented out the lines of /etc/apt/apt.conf.d/99update-notifier and wrote my own updater using bash and zenity and incorporated needsrestart. It’s not pretty but it works.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by Tsiklon on Friday March 31, 2017 @12:41PM (#14005539)
absolutely not, automatically running auto remove may lead to bad things; on occasion autoremove flags other more useful packages for removal.

For example I'm using LVM with my installation on my Ubuntu laptop and after updating the kernel and running "apt autoremove" it removed the LVM package leaving me scratching my head shortly on reboot as to why it wouldn't find my root filesystem (frankly i have no idea how it became "unneeded").

A more sensible approach is how Red Hat do it with YUM/DNF, that is, to allow a certain number of the same packages to be installed, "installonly_limit" in yum.conf. Doing this means that when a new kernel gets installed the oldest is removed to keep the the system at the limit specified.

On my RHEL/CentOS machines I tend to narrowly provision /boot to around 250-500MB. set "installonly_limit" to 2 and the system will keep the most recent kernel and one back. it works for me.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by nominated1 on Friday March 31, 2017 @01:03PM (#14005754)
I see you’re point, though I too use LVM and haven’t seen that happen... weird. I could have been more exact with my response as autoremove does more than just remove old kernels. Anyway, it would be nice to see Canonical resolve this.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by cr0sh on Friday March 31, 2017 @12:19PM (#14005291)
Care to share it? Maybe it could help others...
Re:Ask HN: What do you want to see in Ubuntu 17.10? by nominated1 on Friday March 31, 2017 @12:32PM (#14005435)
I thought about sharing it but like I said it’s not pretty. It involves editing sudoers and holding back config updates for sudoers and update-notifier-common which might cause problems in the future if you’re not aware. I’d much rather see Canonical address it properly.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by Sir_Substance on Friday March 31, 2017 @12:40PM (#14005528)
>Doesn't `apt-get autoremove` remove those old kernels?

Of course it doesn't! Why would you assume such a silly thing? /s https://askubuntu.com/questions/563483/why-doesnt-apt-get-au...

Re:Ask HN: What do you want to see in Ubuntu 17.10? by pmontra on Friday March 31, 2017 @06:55PM (#14009120)
I confirm that it doesn't autoremove. I had to empty /boot on some servers lately.

Anyway sometimes one wants to keep old kernels. I have an old laptop that runs OK with a 3.something kernel and has wierd video sync problems with any newer ones. Ubuntu 16.04 keeps running with that old kernel so I keep booting from that, maybe once or twice per year.

However the proper solution would be pinning a package and autoremoving the others.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by alphapapa on Friday March 31, 2017 @05:45PM (#14008596)
Yes, it does remove old kernels. Read the very link you posted.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by flamedoge on Friday March 31, 2017 @02:56PM (#14006957)
> It's better to err on the side of saving too many kernels than saving too few

But Muh Freedoms! I hate to be subject to one man's opinion of things /s

Re:Ask HN: What do you want to see in Ubuntu 17.10? by Sir_Substance on Friday March 31, 2017 @04:15PM (#14007724)
It's also kind of a garbage argument.

People who know they have broken kernels don't keep upgrading them, they stop and fix them.

People who don't know they have broken kernels also don't know they can boot with an older kernel, so they get nothing from the "backup".

We want to leave some time for people to realize their kernel is broken, so keeping three is probably just fine. Honestly, it would probably be adequate to just bump the oldest one off the queue whenever a newer one is requested. If you've got a tiny boot partition, maybe that means only two revisions. If you've got a huge boot partition it could be 20.

But just keeping them all and making people manually uninstall them gains you nothing, it's user-hostile for no reason.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by bello on Friday March 31, 2017 @12:14PM (#14005240)
Except, it would be nice to keep a few of the (recent) older kernels, in case things go awry with the new update.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by derefr on Friday March 31, 2017 @04:29PM (#14007877)
This already happens: apt autoremove won't remove the package for the running kernel. It'll clean up "old" (N-1 and lower) kernels, but installing kernel N+1 won't allow kernel N to be deleted as long as kernel N is still executing.

Once you reboot/kexec into the N+1 kernel, it'll let you remove the N (now N-1) kernel, bringing you down to one. But at that point you've proven the new kernel works—at least well enough to get to a shell you can run apt autoremove from.

This is why autoremove isn't so auto: if it happened automatically after reboot, it might be running on a now-wedged system (e.g. one that can't bring up the display manager), removing the last-known-good kernel and leaving you with only the broken one.

I think the right middle-ground solution would just be for installing kernel updates to touch a file, and for Desktop Environments to notice that file and trigger a dialog prompt of "you've just rebooted into a new kernel. Everything good?"—where answering "yes" runs apt autoremove. On a wedged system, you can't answer the prompt, so the system won't drop the old kernel. (In other words, just copy the "your display settings were changed. Can you read this?" prompt. It's a great design!)

Re:Ask HN: What do you want to see in Ubuntu 17.10? by fnj on Saturday April 01, 2017 @01:13PM (#14013061)
Fedora/RHEL yum has a much better solution: installonly_limit, defaulting to 3. Kernels which have been updated will only be kept up to this depth. The excess are automatically trimmed during update.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by tripzilch on Saturday April 01, 2017 @11:25AM (#14012548)
Wouldn't a good solution then be to run autoremove before installing a new kernel?

That way, you have kernel N running, first autoremove wipes kernels N-1 and older, then it installs kernel N+1, so that when you reboot into N+1, you'll always have known-good kernel N if it doesn't work.

It's a very similar solution to how a good programmer solves an off-by-one error, doing a shift/rotate shuffle on a for/while loop.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by derefr on Saturday April 01, 2017 @01:47PM (#14013200)
What happens when you have a high-uptime system where you repeatedly "apt dist-upgrade" and end up installing packages for kernels N+1, N+2, N+3, etc., all without rebooting into any of them?

I agree that if the user manually runs an apt [dist-]upgrade—or really any manual apt command—that that's a good time to do apt maintenance work. (Homebrew does maintenance work whenever you invoke it and there haven't been any complaints so far.) But kernels usually get installed automatically, so it can't just run then.

Now, if there was a specific concept of a "last-known good kernel" (imagine, say, the grub package generating+installing a virtual package when you run grub-install, that depends on whatever kernel you specified as your recovery kernel, ensuring it remains around), then your approach could work—you'd always have two kernels, the LKG for a recovery boot, and the newest for a regular boot.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by jhasse on Wednesday April 05, 2017 @02:33PM (#14044413)
Exactly what happens on Fedora.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by joemccall86 on Friday March 31, 2017 @12:47PM (#14005595)
I agree.

I'm running Ubuntu 16.10 currently. A kernel upgrade hosed my setup yesterday, and having an older kernel available saved my butt. I was able to do another `apt-get update` and things eventually worked with the latest kernel.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by unethical_ban on Friday March 31, 2017 @10:41AM (#14004253)
For Ubuntu Desktop, it may make sense for the package manager to keep only the latest 2 or 3 kernels, and automatically purge the rest.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by pbhjpbhj on Friday March 31, 2017 @12:22PM (#14005316)
I had the /boot filling up problem but had thought it was fixed, I'm on 16.04+. I'm pretty sure the last two kernel updates I did removed older kernels leaving me with the current one and previous one ... ?
Re:Ask HN: What do you want to see in Ubuntu 17.10? by kobeya on Friday March 31, 2017 @04:06PM (#14007636)
You can configure apt unattended upgrades to autoremove by default, perhaps you did that?
Re:Ask HN: What do you want to see in Ubuntu 17.10? by cookiecaper on Friday March 31, 2017 @03:02PM (#14007033)
Nope, still doesn't do it without manually invoking autoremove.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by TheCowboy on Friday March 31, 2017 @10:17AM (#14004003)
This is the main problem that keeps me from wanting to set up less technical family members on Ubuntu. It's possible to get in a spot where even a simple command won't solve this.
Re:Ask HN: What do you want to see in Ubuntu 17.10? by benwaffle on Friday March 31, 2017 @11:17AM (#14004662)
Solus uses https://github.com/ikeydoherty/clr-boot-manager now, which purges old kernels and modules, but keeps the modules for the currently running system so HW still works
Re:Ask HN: What do you want to see in Ubuntu 17.10? by rocho on Friday March 31, 2017 @11:50AM (#14004995)
> The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).

What? This is ridiculous and unacceptable. I don't use Ubuntu anymore, can someone tell me what is filling up the boot partition?

I'm currently on ArchLinux and mine is 200MB and it's 14% full! I can't fathom what could occupy so much space.

Re:Ask HN: What do you want to see in Ubuntu 17.10? by pbhjpbhj on Friday March 31, 2017 @12:27PM (#14005376)
It's the way kernel update come in apt. The kernel update is a new package, not an upgrade of a previous kernel package. Thus the old kernels are left in place and the new ones installed alongside. After about 3 kernels have been made available in /boot the previously recommended size for /boot is full and attempted update to a new kernel fails.

It can be manually fixed by removing older kernels ("sudo apt purge ...").

Perhaps I'm mistaken but i thought a fix was in place for this, maybe it was something third-party but apt definitely offered to remove unused kernel package for me recently.