Had they gone ahead with their original plan as scheduled, then 1/3 of those using Android would have encountered certificate errors while attempting to browse Let's Encrypt secured websites. Since then people have moved to newer phones and the number of effected devices has drooped down to about 6%. There is still a bit more than a year left before the transition which will of course continue to move the numbers in a favorable direction.

> then 1/3 of those using Android would have encountered certificate errors while attempting to browse Let's Encrypt secured websites.

It was 50% when they announced the plan initially. :-|

So in 2019, you're saying that LE would have been fine with serving 50% of all Android users certificate errors, is that correct?

If so, what would make them suggest such a plan in the first place?

Initially, they likely didn’t think they had a choice. The root that had cross-signed them was expiring. (And I wonder if anyone else was willing to cross-sign them.) It turns out that root expirations are handled differently (i.e. ignored) on some platforms, including the relevant old Android.

Let's Encrypt is switching to a different root certificate (their own root certificate), which means that extremely outdated devices will no longer be able to access servers using Let's Encrypt certificates.

The biggest concern is old Android devives. By "old", I mean devices using a version of Android from the Obama administration. Android 7.1 is the oldest version that will work, and it was released on October 4th, 2016.

At this point, roughly 6.1% of Android devices are running a version of Android that will be impacted by this change. It's likely that most of these devices are in relatively low-income countries, so depending on your audience this may be much lower or much higher. (If your primary audience isn't in a developing country, it's likely much higher.)

This change will affect you at some point after February 8th, 2024. (Specifically, you'll be affected the first time your LE cert renews after that point. If you're using one of the common clients, it will be up to 60 days after that point.) If you want, you can configure your client to support legacy devices longer, but only up to June 6th, 2024. After then, if you truly need to support very, very old Android devices, you would need to switch from Let's Encrypt to a different (possibly paid) CA.

Only you know for sure if you need to support these very old devices. Most web sites don't.

Man, to think my prior device ran Lollipop (5.x) and it only died (spicy pillow) about two years ago.

> (spicy pillow)

https://www.urbandictionary.com/define.php?term=spicy+pillow

TIL!

This happened to a dell laptop provided by work. After contacting support I had a new battery the next day and the laptop survived.

I had a Fire Phone (Kit-Kat based) that spicy pillowed like 5 years ago. I've had about 4 phones since then though.

Hilarious how 2016 is "*extremely outdated*". Only in tech does anyone think this. :)

Fashion industry would like a word

That explains how they merged.

I have lots of clothes that I wear that are much older than that. And most that are newer than that use designs from long before 2016. They are only newer because the older ones wore out from use. I can only think of a few articles in my closet that could be identified definitively as newer than 2000 by appearance. I imagine this is true for many men, less often for women I suppose.

A lot of perishables (food, medicine, gasoline) get "outdated" pretty fast. Sure, as a concept, lettuce will stay with us for a long time. But an individual Bibb head won't last long, even under perfect conditions.

And yet lettuce can still outlast officials in government!

:) For anyone who doesn't get the reference: https://en.wikipedia.org/wiki/Liz_Truss_lettuce

What is even your point with this comment? Are you somehow claiming that electronics are to be compared to lettuce? This is just absurd.

The very few incumbents that control the modern web infrastructure have (for quite some time now) been intentionally breaking old technology, with absolutely no plans to change this for the future.

It's planned obsolescence, plain and simple. It's just surprising how frequently they obsolete things now. I fully expect them to "train" all the good little consumers out there to replace their phones every year lest most of the web stop working on them.

* * *

The small market share of old device isn't a given. There aren't few of them out there because we, the developers, make them unusable and people have to switch.

Most people don't care about having the latest specs, they want the latest features. If new features only ship to devices that are < 2 years old (like Android updates), older devices will quickly become obsolete. The hardware is still fine - I'm writing this on an almost exact 7 year old OnePlus 3 running a barely year old version of Android and the vast majority of things perform perfectly fine. Even the camera is perfectly fine even for modern standards now that I've switched to a modern camera app by tricking it into thinking my phone is a Pixel.

If we stop making old devices obsolete, dealing with old devices will be a lot easier.

The issue here is software, not hardware. An old Android device would be perfectly capable of running the updated root store if the manufacturers bothered to update it. IE6 also could be switched from, at least theoretically.

> If 6% of Android users globally are on an ancient OS version, how many man-hours would it be sensible for my to spend testing that our systems work for them? How many of those 6% are likely to be paying customers for us?

From a purely financial point of view, I agree. But it stills sucks for the 6% and indirectly this is one of the reasons for electronic waste.

In a way there is a long tail of consumers with smaller incomes and living in countries with weak currencies that benefit from computer hardware and software and services that are developed targeting wealthier consumers but still have to deal with the fact they aren't really the target market.

Isn't the EU advocating (or legislating) longer life spans for consumer electronics?

If so, I hope they're covering both hardware and software and configs.

That certs eventually expire is obvious to me now that someone raised the issue. But I wouldn't have foreseen the problem on my own. So I imagine policy makers will need to be informed too.

> Isn't the EU advocating (or legislating) longer life spans for consumer electronics?

Yes, there is legislation for this that mandates a number of OS updates and security updates. I'm hoping Android manufacturers will stop shovelling new SKUs and concentrate on a few they can actually keep up to date.

It depends; there’s also fashion or entertainment; in some aspects also education, law and I’m sure many others.

Compare a 2016-era phone with the current equivalent (e.g. iPhone 7 vs iPhone 14) and there is a huge difference, many of the specs. are 4x those of the older model.

Don't get me wrong, I really don't like the whole 'disposable hardware' culture, but you can't deny there has been a huge leap in tech over the last ~7 years.

There are computing devices that are not mobile phones or laptops that also depend on encrypted connections. Usually these devices last a lot longer.

But usually these other devices can also update their software, it is quite unique for the android operating system to run old and unupdatable software.

Or they should if they are web facing. For security reasons at least.

If your whatever device doesn't support a root CA from 2016, chances you should not let it connect to anything over the internet. Most bad software comes from "hardware" vendors, who think they can handle them the same as any other physical part. Stop blaming software people.

I don't agree, I have an iPad Mini4 from 2015, and that's still good for most day to day tasks (although it feels slower than in the beginning thanks to iOS updates being less optimized).

Same with my mid-2014 13" MBP. While my M1 MBP is a lot faster for things like building complex C++ code bases, for stuff like web browsing the old Intel Mac doesn't feel much different.

Compared to the incredible hardware progress between 1995 and 2005 (where a 2x increase per year was quite normal) it definitely feels like we're on a plateau since around 2010.

It is not a matter of the age of your hardware but your software. Ipad mini 4 got its latest OS update in 2023. The problem is that in the Android ecosystem it is unfortunately common to only have a short window of OS updates from the release date of the hardware.

Yes, but parent was specifically talking about hardware specs. And a 4x times increase since 2016 isn't much progress compared to the 'good ole times' ;)

False, I can use my 12 years old Android perfectly. While I had to ditch my old iPhone 7 (!) because I couldn't install most apps anymore.

iPhone 7 is still a perfectly usable device but bricked now.. Stuck in iOS.

Would you care to elaborate which device it is? 12 years old means it at the most could have come with Honeycomb when you bought it. Has the base OS been updated since? It is great that you bought a device which presumably got OS updates for many years, unfortunately that is not common for android devices. And the problem is, if it did not get an OS update, the truststore did not get updated, and thus it does not trust the LetsEncrypt root CA sometime next year. Except for Firefox which luckily comes with its own root certificate list.

Iphone 7 got an iOS update to 15.7.7 in 2023 and its truststore contains the ISRG Root X1 certificate: https://support.apple.com/en-gb/HT212773 I am unsure which apps you cannot install but a quick look in the app store indicates that Zoom, LinkedIn, Notability, 1Password, Disney+ and Netflix all support iOS 15.7. And in my anecdotal experience I could find no app with a minimum OS requirement greater than 15.7. As far as I can tell you need to go back to iPhone 4S to find an iphone which does not support the LetsEncrypt root certificate. That device is only 11 years old, so still worse than your Android device. And I do not think there is a workaround by using a different browser like for Android.

My grandmother has a Google TV, still works with a bit of stuff. Running Honeycomb.

Interesting - what apps were working as new installs on 12 year old android but not iPhone 7? That’s unusual. iPhones update cycle is much longer and they usually ship w a recent iOS version when released

I have an iPad Pro (1st gen, 2015) that I continue to use as a (secondary) daily driver.

My laptops are both 2015 MacBook Pros.

I’m sure there’s plenty of use cases where the gains are valuable. If you’re a photographer, developer, gamer, or web browser without an Adblocker, I’m sure it’s immense. For me personally, the gains are questionable.

I'd say you're nearing the cliff.

I have an iPad mini that's only a year or two older than that and software upgrades aren't supported, nor are there many (any?) apps available for that version of iOS

My mother has an iPhone 7. She just upgraded from an iPhone 5.

It works perfectly fine for her, it runs the two apps that she uses: Camera and WhatsApp. And the price was reasonable, only half of her weekly pension. There’s no way she could afford an iPhone 14, that’s a whole month’s income.

She doesn’t know the specs of her phone, or care.

I still have an iPhone 7. With my usage, the battery lasts 1 or 2 days. I can read the occasional email, my newspaper, use Teams and Telegram when I'm not behind my laptop. It even plays games. What more do I want?

My mom uses an iPhone 7. She took video of us brining out the cake to my child’s birthday. Let me just say, I won’t make that mistake again. The video quality is terrible.

> She took video of us brining out the cake to my child’s birthday. Let me just say, I won’t make that mistake again. The video quality is terrible.

Not to be too salty but that cake must have tasted worse than the video looked terrible.

Yes, I too followed your salty comment down that mental rabbit hole.

I'm with you, apple stans, but it's worth pointing out it's not the apple devices we're talking about here. That iPhone 7 can run iOS 15, which is out of date now, but only by two years. No letsencrypt certificate troubles.

Depends on how you define "out of date". Is iOS 15 getting new features? Nope. Can the iPhone 7 get iOS16 or newer? Nope. However Apple are still releasing security updates for iOS15, the latest update for iOS15 was just 20 days ago - https://ipsw.me/15.7.7

EDIT: However, I don't suspect that iOS15 will be supported for much longer, iOS14 last update was back in Oct 2021.

And apple supports iPhones much better. Both iPhone 5 and 7 I believe have the isrg root now. My wife likes older iPhones for weight / size reasons and the longevity of these is impressive especially if you pay for a battery swap

I don't think devices have progressed much in the last 5-7 years. Before that, computers became almost twice as fast yearly. So yes I'm actually denying this. Devices haven't changed much in the last 7 years.

The only thing that feels outdated on a 7 year old work laptop that I'm using is the amount of memory.

This issue doesn’t apply to the iPhone 7 yet given that that device received the iOS 15 update in 2021.

For all the talk this “disposable hardware” and “planned obsolescence” issue is much more a problem on Android.

OEMs could update the cert store on those old Androids. It's a cheap phone problem.

I'm still using my LG G5 from 2016. On my 4th battery right now and will keep on using it until it dies. Still runs smooth browsing the web or even playing brand new 3d games. Batteries are 10-20eur on ebay. The cameras lack nothing and it unlocks with a finger print scanner on the back. This means that in the past 10 years I've spent about 550EUR on mobile phones.

About a year ago a friend of mine complimented me on a new phone, until I told him that it's my old LG, I just stopped using the case. He seemed annoyed after that. Must be because I got funny looks and giggles years back when I said I'll never buy a phone without a replaceable battery.

The leap in the Android ecosystem is far less significant

There are many low cost Android devices about whos processing power hasn’t really improved much over the same timespan

And although iPhones have got much better it’s worth remembering how much their price has increased too (SE excepted)

Auto manufacturers are required by US Federal Law to have parts available for any models they sold for the period of 10 years.

Something like this is needed for mobile phones, and probably other electronics as well.

My current phone is a pixel 2 released in 2017. It's perfectly great and there's no reason it couldn't continue receiving updates other than planned obsolescence.

This "huge" difference is only at the top. If anything, the software has gotten worse... my parents hate the "new" gestures... if they didn't give the option for the old bottom row buttons it would be doubly worse.

I know people who refuse to watch movies that are older than a couple of years.

It is in tech terms. Just deal with it.

Phones from 2016 are ancient and you can buy newer phones dirt cheap. The Samsung A series costs almost nothing, same with all the cheap Redmi phones and Oppo and whatnot.

You can do it but that doesn’t mean you should.

Throwing away a phone whose hardware may be perfectly working, but its manufacturer decided to be too cheap to use anything but walled-garden components, feels like an utter waste of resources.

(By walled-garden components, I mean chips whose suppliers make it difficult or impossible to develop an open-source driver.)

If you really need that old phone, use Firefox Mobile which has its own root store.

But good luck using it

This might be less of an option then you'd think. Android software is often compiled assuming the CPU has comparatively recent instruction set extensions. Older devices, especially lower-end devices (which are more likely to be stuck on older software and more likely to be owned by people without the resources to frequently upgrade), might lack those extensions, making the app unusable. This is not a hypothetical. I've experienced this with FF a couple years ago (and even spent a couple hours trying to figure out how to build from source, but ultimately gave up.)

FWIW, I use Firefox on a 10 year old phone with Android 5.1, and it runs just fine. The phone is built on an x86 Atom SoC, though.

https://www.intel.com/content/www/us/en/products/sku/75203/i...

What extensions other than NEON are you thinking of? Neon was made a CDD requirement for Android 6.0, released on 2015. Even before that it was almost ubiquitous. The last shipping devices in any volume missing NEON were Tegra 2 tablets from 2011.

That "almost nothing" is a typical blue collar monthly salary in my shitty town.

The typical tax-related depreciation term for computer hardware is 5 years over here. Which is already half of what typical office equipment is expected to last. After which it is usually resold or given away to goodwill. 2016 is just 7 years back, so there will quite certainly mobile phone where the tax writeoff period isn't even through which are affected by this. Truly hilarious!

Its not even 10 years old. The hardware is aging but the only reason its “outdated” is due to a software lifecycle forced by one or more corporate entities. These are an obvious tool to force sales/upgrades.

I’d be nice if this community weren’t so quick to defend planned obsolescence, especially at a 2-3 year pace.

Note that in this particular case you can still use alternative browsers that come with updated root CAs.

This will still break anything that uses a system webview.

And also any apps that make API calls over HTTPS, regardless of webview, right? I believe they cannot use HTTP, only HTTPS. However, I'm unsure whether they can disable certificate verification. Presumably not, for the same reason...

I think it was apple that forced the use of HTTPS in app APIs, not Google.

I don't believe android apps are locked to HTTPS as a rule, though obviously a lot of them likely don't have fallbacks.

Be nice if government out in laws to force at least a decade of "consistent operations" (it works as well at year 9 as year 1).

Theres really no reason that can't be done, it's not like phones have the amount of variety as PCs and MS and Linux have both managed to handle some much older hardware.

When talking about security updates it most certainly is. You can’t blame LE for that. That responsibilty lies squarely on manufacturers who are not even maintaining the bare minimum of security patching like an up-to-date root cert list.

I just checked and my own phone is affected by this. Coincidentally I just started getting the "everything feels so slow" thing that seems to inevitably happen to old devices.

That is even worse when you consider that tech brands often design their full lineups with the buying-power of rich countries in mind, their lowest option each year often being way too expensive to the lower classes on developing or poor countries, so there they often advertise previous years' models as the official budget option (often two, three or four years behind). As result that "2016" is more like 2018 in many countries, that is, someone may have walked into a reputable store, bought a "brand new" phone in 2018, and then it will be unable to access those websites just 5 years later.

I dunno, mayonnaise in my fridge from 2016 is probably extremely outdated.

No no, that's just cave aged mayonnaise, and it sells for a premium. Throw in a blurb about hand crafted artisan, and it'll be a rocket to the moon looking sales chart. You just gotta know how to sell it. These aren't outdated Android devices, they're vintage legacy devices.

I was walking round our Hq last week and noticed some monitoring screens. Running the code I wrote back in 2005. Code that is now old enough to drink.

Still works.

I agree. Personally, nothing from 2016 counts as "extremely outdated" to me.

It's not that 2016 devices are outdated per se, as a counter point, see the iPhone 7, which runs now on iOS 15.x from 2021, but still receives updates [1].

It is cheap Android handsets stuck on Android 6 Marshmallow or worse that never have received an OS upgrade.

[1]: https://support.apple.com/en-us/HT213811

I think you could manually substitute in the old chain—up until the point the signature expires. That would get you until the September date. I wouldn’t recommend it, but that could be useful to buy just a little more time if one was desperate.

A LOT (like, most of them) of commercial Android simulators use Android 7.1 (some even Android 5). Granted most people usually only use them to play games, but it's great that at least this aspect is covered.

Where do low level (eg Android platform) devs pull root certificates from? Who governs that list?

Android itself (I.e. Google) maintain thier own list; they manage it well IMHO but made a catastrofically bad desision to never update it outside OS versions (they have fixed this in the next release).

In principle whoever is building the Android system can choose whatever roots they want. In practice the Android Project is a Google project, and follows Google's policies.

Also in practice, Google's policies strongly resemble Mozilla's. Mozilla's root programme is overseen by mozilla.dev.security.policy, a public group. So, to some extent the answer to "Who governs that list?" is you do, much as (if you live in a democracy) it's your fault that your government is terrible. You could work hard to improve things. But, you probably won't.

Thanks for the summary. Are there paid CAs that provide tooling like certbot (or work with it)? I don't particularly care about LE or it being free but I like the tools and people using old devices who are easily scared by warnings should be able to use my website...

The protocol used by Certbot is called ACME, and is supported by a bunch of CAs, both free and not.

In addition to LE, there’s other CAs like sectigo, digicert, and Google Trust Services that support it. A few are listed here: https://www.acmeisuptime.com/#CA-Software

I'm disappointed that we can't come up with a way for encryption to work forever.

Like, if I find a knife from 150 years ago, it still cuts cheese. Sure - it isn't the most modern tech, but it still does the job it was made for. Even though cheese recipes have changed, they are still compatible with an old knife.

Yet a phone in 150 years will be 100% useless. Not only useless because other things have moved on, or because the hardware has degraded, but useless by design because root certificates expire.

We shouldn't be putting anything into consumer electronics with an expiry date.

A knife is an extremely simple tool and you're making a very modest requirement of it (that it should cut cheese). There's no way that a 2023 smartphone is equivalently simple, nor that "Just access arbitrary remote stuff over the Internet" is a similarly modest requirement.

Lets try something appreciably more complex and do an equivalently tricky task. Lets use an 1873 steam train to travel from London to Derby.

We run into more or less the same problem. Functionally, this can work, you'll have to buy the coal from somewhere and hire tankers to move the water where it's needed since there is no longer provision for that- but much harder you'll need to secure an extraordinary amount of special case paperwork to make this happen, because obviously your 1873 steam train isn't authorized to use this route, you will need to retro-fit safety equipment that an 1873 steam train was never designed to work with, and you will need to re-train people to operate it.

Don't worry though, you have preserved the important part - your 150 year old train is still ludicrously unsafe compared to its modern equivalent like the phone. Relatively minor impacts will turn the passenger compartments into kindling, with passengers still inside, because they're basically just a wooden box resting on a simple metal platform, not a monocoque design and even after fitting mandatory safety equipment the train can't effectively stop anywhere close to fast as modern trains if things go wrong.

Have you ever tried accessing the modern web with a Windows 95 computer using Netscape Navigator?

Even if that computer had up-to-date certificates, basically nothing about it would be compatible with the modern internet. It's not just a question of of "expiry dates", it's that you cannot make any forward progress if you have to maintain backwards compatibility forever.

Internet protocols, file formats, hardware standards... all these things need to be able to change over time. Hell, with the old phone example, your biggest problem with a sufficiently old phone is whether it can even connect to the network - I recently tried using a retro Nokia as a backup phone, but realised that the SIM card I'd put in it was for a network which didn't offer 2G coverage. Phone couldn't even make calls or send texts because it wasn't 3G compatible.

Yes, and it worked hilariously well, because we've worked on bridge technologies for things like this: whether it's local proxies to strip or modify SSL or even just render an entire page as an image and transport it to the old browser as-is.

Solutions will come and go. Hell, what's stopping someone from building an on-device VPN that does the SSL translation itself?

> Have you ever tried accessing the modern web with a Windows 95 computer using Netscape Navigator

Well, not exactly web browsing, but somewhat using API... yes :) https://www.dialup.net/wingpt/tls.html

Not that it would be necessary

> We shouldn't be putting anything into consumer electronics with an expiry date.

So you wouldn't have half the consumer electronics we have because nobody would be able to afford them.

We should be making consumer electronics with reasonable life expectancies. 150 years in the future I don't want to use something from 150 years ago that "still works", I want something that is new with more capabilities, that is cheaper, that does the thing better. Why create undestructible phones that will last 150 years when nobody other than vintage collectors will want them after ~10 years due to other problems with material degradation, fashion etc.

The optimal point is to not create discardable things, but it's also realising that if you're making something forever you're going to use up way more resources and people will still stop using those devices for other reasons, so you just created more waste.

Too short life = too much waste due to replacement needs.

Too long life = too many wasted resources per-device which will be abandoned for other reasons.

> 150 years in the future I don't want to use something from 150 years ago that "still works", I want something that is new with more capabilities

In 150 years you could as well not have anything new on par with the currently available devices.

Last night I used some 100+ year old tools to poke a hole in something. But I almost bought a $10 awl last time I was in the store because the old tools aren't really fit for purpose, and they're really just decorations.

The encryption itself will work "forever." (Realistically, very powerful computers could some day obviate contemporary encryption).

The tricky part about PKI and signed TLS certificates has little to do with the encryption and a lot to do with networks of trusted parties. People die and are born, relationships evolve, and so these networks are inherently dynamic.

Certificate expiration is a feature by design, and it's included for a reason.

Any solutions for the problem would still only exist in new devices. And the announcement does indeed mention that new devices will be able to update their certificate store.

Is it possible to add new Let's Encrypt root certificate to old android device manually?

Because the root issue is just that — no root certificate in the system.

You can add a user certificate, but in pretty sure it's up to the app if it wants to trust that certificate.

> but useless by design because root certificates expire

Should be noted that the new design (Android 14) is to allow certificate updates to be separate from operating system updates. Not sure if this is done via a standardized API or something proprietary, so the problem of servers going down forever might remain.

So even encryption encourages planned obsolescence!

You can add a root certificate to Android, but if you're not a superuser (haven't "rooted" your phone, different usage) you can only add it as a "user" certificate, not a "system" certificate. After I believe Android 7 apps will by default not trust "user" certificates, so this would not help you, but since newer devices should include the LE root cert this just might be a viable solution. It's still not perfect. It's confusing and (probably by design) somewhat difficult for a lay user to do. And Android will give you a scary message every time it starts, saying your phone may be compromised.