I agree that things could have been done better in many ways. However, as explained, it is a legacy application, which does not see any active development nowadays and it would just not make sense to build such a robust QA. This particular app have been written long time ago by another company and there's not even simple unit tests. That's the hard truth. But still, even as complex setup as you have, there's still going to happen mistakes and the real problem is that there is no way to pull-back/cancel/rollback release.

How would staged roll-out help in this situation for all customers? When end-user gets the faulty version of the app, does he/she have a way of getting the non-faulty version somehow?

Unfortunately, as you noted, the game is rigged and you have to play within the sandbox provided. With that, and as you also noted, your testing simply needs to be better, especially since this is a customer-facing app and not an app that's solely used internally. As for dealing with the rollout when a serious bug is now in production, if you didn't roll out your app to 100% of the user base you could halt the roll out so it wouldn't affect any more customers. Then, while far from ideal, you could ask affected customers to uninstall the app and then reinstall it, and the newly installed version would reflect the previous version of your app.

A-ha - good tip about roll-out, uninstall & install. Was not aware of that possibility. But yeah, it's still a hack and there should be a better way - just let me delete/cancel latest release even if roll-out is 100% for whatever reasons.

About better testing - there's always room to improve testing, but no way it's going to happen with a legacy application where no active team is assigned. Only these irregular updates mainly forced by Google are done. Unfortunately.

building a business on any big vendor (google, apple, amazon, microsoft, etc) without at least making sure these risks are shared with the client seems like a fast way to run into these costly overtime problems.

Rigged seems a bit extreme. They had to make a few changes to keep things up to date. That they tripped over this small hurdle and went tumbling down the stairs doesnt change that it's a small hurdle.

Perhaps. I meant it more for the Google Play Store policies as a whole, and they are voluminous, and not just this specific case.

> it would just not make sense to build such a robust QA.

Well that's a lesson you won't need to learn twice, isn't it?

It's true that in the future I would do some more thorough testing, but never-ever for this legacy application can I build a bullet-proof automated testing solution - there will not be a budget for that for sure. However, even with a fancy solution mistakes will happen and you still can't stop release propagating. It's just a matter of time when it happens.

"A bullet-proof automated testing solution" isn't needed for the type of problem you describe. Anything that logs in (including a human on a real device) would catch it even if it does nothing else.

Exactly. It’s a poor excuse that the author didn’t even try to log in to the app with the newest version.

Android as an OS does not support rollbacks. It can only upgrade apps in monotonically increasing `versionCode`s. The simple reason is that client side data may have been upgraded by a newer release to a format that is now incompatible with the old version. Sqlite databases follow the same policy.

This is well-known to anyone who has been doing Android development for a while.

Totally understand that a roll-back is more complex, but it doesn't explain why pulling back/pausing/deleting/yanking a release is not possible either - customers who got the newest faulty release can uninstall & install to get the previous version back (no need to worry about incompatibilities here) and the ones who did not get the updated version yet can stay using the old one until a new fixed version will be released.

This is exactly what staged rollouts are for. The author ignored that best practice. You can halt any release as long as it is in progress. Once you mark it complete (defined as rolled out to 100%), you can’t roll it back.

Each track in Google Play (alpha, beta, canary, prod) can hold up to one release at a time. It would get really confusing if it allowed more than one. And with the other rollout safeguards provided decelopers, it’s quite possible and very easy to do exactly what you’re asking for.

"may" is the reason why not supporting rollbacks at the OS level is baddesign

> and it would just not make sense to build such a robust QA

Well since it broke, it kinda of would have made sense.

> This particular app have been written long time ago by another company and there's not even simple unit tests

He didn’t do a simple smoke test.

Just to add, even if you do not change anything at all, you will spend many hours just getting the build environment up to snuff if it hasn't been touched in over a year. Gradle, the plugins (android, crashlytics, etc), the build.gradle DSL, dependencies ... its quick bit rot.

Protip: that field takes decimals too. We always go for 99.99999999% so no users get left out accidentally. But it allows you to cancel a rollout like you say

That's hilariously crazy.

What you're saying is totally true and the only pragmatic way to update/release. But it is a shame that if the api update compiles you can't expect things will just work the same way.

This is why you set a version number for the API you want. No type system in wide production use is able to encode the semantics of APIs sufficiently. Many of the best APIs out there have version numbers that allow you to pin the API structure and semantics – GitHub and Stripe come to mind with this.

There’s this company out of I think Washington state who builds an operating system for x86 and x64 systems. I know for a fact that GUI applications from 15 years ago still work on the modern version. It’s called Microsoft Windows. You should check it out. The win32 api has changed but it’s backwards compatible.

Windows has "Compatibility Mode" to support these things. Additionally there are layers and layers of Windows API – an app built against Windows XP may still work but it's not using the same tech as one built for the current version of Windows. This is part of the reason that users always had to install different versions of .NET, or part of the reason why Windows is a huge OS (API surface area, disk usage) in a way that isn't suitable for mobile devices.

Yes MS are better at backwards compatibility, but everything is a tradeoff and they receive plenty of criticism for the direction they take.

Desktop computing has different constraints and is moving much more slowly than mobile computing, so it's more feasible to maintain that compatibility.

> This a very major mistake and the reason why we keep 11 virtual machines around with all the versions of Android that our app is supported on.

Which is fine, if that's what you have to do, but at the end of the day I really do wonder what their 30% cut of your revenue is for, then.

The section that's quoted here made me briefly wonder if GP was satire. As someone with no Android Dev experience: WTF Google?!

This feels a little victim blaming. Those two things shouldn't be an issue at all.

This is my same sentiment, author has been too dramatic with the blog title. Updating from 30 to 33 SDK is just a disaster waiting to happen. In the mobile world that is just too long.

But this is also kind of a problem created by Google - why force from 30 to 33 and not force from 30 to 31, then 31 to 32 etc? Gradual API level upgrades would make more sense in my world and probably would be less risky for every party.