... perhaps if we were to actually get serious about doing small/incremental updates all the time, we'd get better at actually doing them without incident.
I think one of the major obstactles is that the software stack has become ridiculously (and hideously) complex over time. If we could converge towards a world more like what's described here[1], then I think we'd be in a better place overall. (Granted, this the video is talking about deploying services, but AFAICT there's little essential difference between that and a modern GUI system with all its DBUS interfaces and whatnot.)
So I get in the habit of upgrading the dependencies of all the apps I work on, every time I work on them. Issues happen, but only to one dependency at a time. It's manageable.
What I would love is to eventually have a CI server do it for me. Every single day, it would run bundle update, run the tests, and deploy unless there's a problem. If there is a problem, it drops me an email with the trace and fixing it becomes part of my morning routine.
If subtler problems surface this way, then I've discovered an oversight in my test coverage, or an overly complicated architecture that I need to remove dependencies from.
I'll probably implement this sometime this year. I'm thinking I'll want to redo deployment instead of relying on Capistrano, then finally growing my own CI solution. I'm slowly moving away from big monolithic apps to smaller, homegrown solutions that do only what I want them to do. I've already reimplemented provisioning and configuration management. I believe in DevOps as code.
Agreed. A while back, there was a security fix in the default php5-fpm configuration (0666 -> 0660, I believe), and I skipped it when updating. My website went down until I learned to change the owner of the process. And this wasn't even a dist-upgrade.
Sometimes manual intervention is required.
In my case, I run updates almost every day. If I had to wade through 6 months of backlogged updates, I would have wasted a lot more time identifying where it broke. Smaller feedback loops are a win.
I realize that this is a nontrivial proposition, but if we're ever going to realize the dream of "interchangeable parts" we're going to have to learn to do this stuff.
That would require more developers to actually give a fuck about the user experience not breaking from release to release.
(I'm not very well placed in the software cycle to actually start this feedback loop, but one can wish, right?)
What I like about this article is it's talking about what the global optimum is -- what's the best approach for the entire ecosystem? Not just for one individual or one org. What's Pareto optimal? (Or maybe it's a little less general -- what's the best approach for those who depend on FOSS, and don't pay someone like RedHat for long-term support and CYA insurance).
And basically I agree. The world's upgrade cycles could stand to be a whole lot shorter than they are. Every time I think about all the hardware in this world that'll spend its whole lifetime vulnerable to heartbleed, or shellshock, or a gazillion smaller vulns, my skin crawls.
That sounds painful.
I think the author is really advocating for a faster feedback cycle. Imagine if instead of a big upgrade every X months, you had a tiny upgrade every day, which didn't require rebooting and happened in the background.
Breaking changes could surface the same day they were shipped and be rolled back or fixed.
Now imagine that conservative players also constantly upgrade, but always stay 6 months behind the bleeding edge, getting only releases marked as "tried and true". Eg, if something had to be rolled back or patched, they skip over the bad version.
Wouldn't that make upgrades much less painful for companies like you describe?
Now this does mean that you have to make very sure that the upgrade you start at 4pm on Friday will be totally done and bug free by 9:30am on Monday, of course.
And I agree with your larger point that in a situation like that testing done by someone else in a different configuration is of limited (though nonzero) utility.
What I'm about to say has been said a thousand times, but I guess we need to keep saying it as long as this myth survives. Are you using PPAs? Are you using PPAs that provide newer versions of software that also ships with the system? Are you using third-party drivers (looking at you, Nvidia)? If so, there's your problem. It's not that difficult to blast away your PPAs with "ppa-purge" prior to running a distribution upgrade. As for regular package updates - "apt-get upgrade" won't break a system, that's not been already severely broken by the user. Period.
I use Ubuntu on the desktop and the server, and have done many upgrades over many versions. I don't recall a post-upgrade broken system in recent years that wasn't due to third-party stuff. Even then, it's trivial to fix - reboot to recovery mode if you can't get your X11 on, "apt-get purge" the offending stuff, reboot and carry on.
My impression is that Ubuntu screwed up on a major version upgrade several years ago, and their upgrade process is still being dinged by folks due to the wave of bad publicity that that incident created. Which is a shame, because Ubuntu post 12.04 is a rock-solid operating system that is a joy to use (upgrading included). Granted, third-party packages can sometimes reduce the overall stability, but then what do you expect? It helps to carefully vet which PPAs you use - PPAs that are done by people who know what they're doing don't tend to cause trouble.
If you want a rock-solid system, do a clean install of an LTS release, don't set up PPAs, and enjoy. I have a side Thinkpad with 12.04 on it which is exactly that, and it's the most stable system I've ever used. And it's being kept current with security patches and bugfixes with regular "apt-get upgrade".
Not that it's relevant, but to respond to your first point: you know people who roll their browser, and a few Linux distros roll their release too. I've heard of some startups who use a Fedora rather than CentOS for that reason.
Ubuntu and Fedora releases happen already in 6 month cycles. This is not some novel unexplored concept.
Note, that I do not mean to diminish the accomplishment. Especially as a fan of TeX and someone that often toutes that maybe we are learning the wrong software lessons by constantly rewriting everything to be in the latest safe language/paradigm.
However, to not acknowledge all of the progress that is made in other areas is just as dangerous as not acknowledging the benefits of more deliberate methodologies.
It probably helped that the complexity was low- I remember the default install occupying some 300MB of disk and a few tens of MB of memory, in a time when Ubuntu already occupied some 10GB.
That was more than half a decade ago though.
This works because I quit caring about the UI experience. I accept that when I upgrade, things will be dramatically different.
If I adopt a similar policy on my computer, I get similar results. However, as soon as I start actually caring about more and more data on the machine, things get tricky.
And this is where a lot of customizable user interfaces hit ridiculous trouble. Those customizations are rarely treated as important data. Worse, there is often a lack of ability to clean up excess preferences.
Which leads me back to my main point. There have been a lot of advances in consumer user applications that are easy to ignore in server applications. Which makes upgrading much more difficult in the consumer space.
But you are right about the lower "data risk" on phones, thanks to cloud services and such.
Tesla cars get automated over-the-air updates far more frequently than every 6 months.
You can. Think AWS systems (as an example) where you don't log in and patch, you spin up a new instance instead. This can't apply to everything (stock exchanges, MRIs, etc), but for the software affected by glibc, absolutely.
Most new computers come with a warranty that is 90 days. Why should software support be 5x longer?
Where? In the UK, consumer protection laws would probably force a minimum of two years in most cases. Even after that, if an expensive device mysteriously failed on day 2x365+1 in a way that wasn't reasonably expected, the purchaser would still have some relevant rights.
Why should software support be 5x longer?
It is all about reasonable expectations. No-one paying for basic system software would reasonably expect it to stop working after three months.
Of course if we're talking about Open Source software that you're getting for free then it's your own responsibility to make sure it does the job you need and no-one owes you anything. This is part of the reason that selling support for otherwise free software is a viable, and sometimes highly lucrative, business model.
Wow, I'm in very strong disagreement with the author on this point.
I don't think I could find a single person, technical or otherwise who didn't use the same operating system for more than a year. In fact its the default for almost everyone who isn't technically inclined to upgrade their os only when they get a new computer.
What about every single car produced today. I doubt you'd want to upgrade them every 6 months.
As a ubuntu user, I find that I cross my fingers every time I run sudo apt-get upgrade. About half the time I get broken builds and once a year my OS will just flat out crash from it, or fail to reboot in virtual box.
This view, while idealistic is just so laughable I can't begin to believe the author is serious.
I've worked with some mission critical systems( stock exchange's) They take more than a year planning a big OS upgrade. To turn around and tell them that they should be doing this every 6 months is just so out of touch with reality, I don't know how to respond to the author:(
Many companies won't install a new piece of software until its been proved out in production for 6 months to a year by someone else....