Firefox ships ~100 binaries for 100 different targets, from mobile phones, to desktops, tablets, ... x86, arm, sparc, ppc, riscv, ... X linux, windows, macos, freebds, openbsd, netbsd, .... X arch linux, debian, ubuntu ..

At some point you need to say this is what we want to ship, and branch, and make sure that not only all tests pass, but that the installers work, run the benchmark suite that might not be run on every PR, make sure the updates work on all systems, appstores, etc. Accept new patches to fix bugs, etc.

I suppose they might have a release team that does nothing but releases, and if creating a full release takes a week (no idea how long it takes honestly), then maybe they could release each week.

But I don't think they can release a new version every day, because probably running their whole CI takes longer.

E.g. before shipping a version of Rust, the RC is used to build and run tests of all packages in the Rust package repository. This takes 4 full days. One could probably throw money at the problem and speed it up, but at least right now, releases just cannot technically happen faster, unless you are willing to compromise on quality.

Right. Desktop software is also much more difficult to QA than server software because of the lack of control over the environment and the difficulty of rollback.

Also, for desktop software, I have found that small releases are a lot less risky.

You experience small issues, one by one, instead of a massive amount of bugs coming all of a sudden.

For example, last time I updated a Linux distribution with 6-month releases this was the case. Whereas with a rolling release one, it's just much more pleasant.

This is true of releasing code in general. Changes that accumulate over time actually increase risk whether it be desktop or server-side.

Not to mention, different builds per language for each OS. [1] This has never really made sense to me, but I'm sure there must be a good reason.

[1] - https://ftp.mozilla.org/pub/firefox/releases/68.1.0esr/linux...

My educated guess is that the localization data for every language is pretty massive, so per-region downloads can include the text for their region without dragging along a few extra megabytes of text/images for other regions. If you swap regions after installing it can probably download on-demand.

Historically reducing your download size by a couple megabytes has had measurable positive impact on conversion rates (from webpage visit -> install) so I expect this tradeoff made a lot of sense in the past even if it's less important now.

EDIT: One other thing worth note is that there's zh-CN and zh-TW, and I can imagine the Chinese government being very irritated about the idea of their release having separate Taiwan loc text/images baked in since according to them Taiwan doesn't exist. Major software like Windows has already had CN-specific releases for this (among other) reason(s).

`zh-CN` and `zh-TW` are generally just old-school ways to say "simplified Chinese" and "Traditional Chinese" (two similar writing systems). The confusion comes in as the Chinese language family has multiple dialects and variants, and there are multiple writing systems.

The current practice is really a dimension-reduction, where a N×2 matrix [(CN, TW, HK, SG, MY...) × (simplified, traditional)] collapsed into a 2×1 vector (TW, CN).

The language labels should have been `zh-hans` and `zh-hant` if one means to differentiate the writing systems, and not the underlying linguistic variants.

According to Beijing government, Taiwan absolutely exists, it's just that it is not a sovereign state. Hong Kong has been handed over to China for more than twenty years and people in Hong Kong continue to use zh-HK – Beijing government seems to be OK with that.

zh-CN and zh-TW has nothing to do with that, and I'm virtually certain there are no "Taiwan images". in applications which provide zh-CN and zh-TW, including Firefox, the former means Simplified Chinese and the latter means Traditional Chinese. theoretically, it could mean Taiwanese Chinese, but then there would also be zh-HK, zh-MO, zh-SG...

They build downloadable binaries every 12 hours, called Firefox nightly, which is to say the bottleneck for full releases isn't building or automated testing. Mind you, the nightlies are experimental, and not for general use.

Building is in general just a small part of the release pipeline.

Mozilla only provides binaries for Linux 32-bit, Linux 64-bit, macOS, win32, and win64:

http://ftp.mozilla.org/pub/firefox/releases/69.0/

All of the various Linux distributions just repackage those binaries. iOS and Android are distributed only through the App Store and Google Play.

The BSDs and other ("tier 3") platforms have to compile from source and distribute their own packages:

https://developer.mozilla.org/en-US/docs/Mozilla/Supported_b...

> All of the various Linux distributions just repackage those binaries.

Source please.

Never mind, it's simply wrong so don't even worry about it!

Even if you’re right and they’re wrong, the snark was unwarranted IMO. You should either correct the GP or request more information (like a source), not ask rhetorically.

Gentoo does it for firefox-bin:

/usr/portage/www-client/firefox-bin/firefox-bin-69.0.ebuild

    MOZ_HTTP_URI="https://archive.mozilla.org/pub/mozilla.org/${MOZ_PN}/releases/"
    ...
    SRC_URI="${SRC_URI}
        amd64? ( ${MOZ_HTTP_URI%/}/${MOZ_PV}/linux-x86_64/en-US/${MOZ_P}.tar.bz2 -> ${PN}_x86_64-${PV}.tar.bz2 )
        x86? ( ${MOZ_HTTP_URI%/}/${MOZ_PV}/linux-i686/en-US/${MOZ_P}.tar.bz2 -> ${PN}_i686-${PV}.tar.bz2 )"

Doesn’t gentoo essentially do that for (most?) everything?

Only massive packages (firefox, chromium, KDE, Gnome, etc.). The entire promise of gentoo is the opposite: fine-tuning and local ~~heater~~ compilation.

Most linuces build their own firefoxes from source, same as the BSDs.

it mostly was the case to remove the logo at some point in time :/

> All of the various Linux distributions just repackage those binaries

Debian and Ubuntu don’t build from sources?

They do.

You can notice by looking at the build time in the version string.

This makes it impossible to share a profile between two GNU/Linux distributions (or even the official binary and the distribution binary) without hacking with the current versions, even if they are effectively the same version: one will complain that the profile has been run with a more recent version.

(But I sure expect and want my distributions to rebuild Firefox - I trust Debian for making sure no bit of proprietary code is here, and makes tracking opt-in, more than Mozilla)

Although this comment was already debunked, I'll add Arch's Firefox PKGBUILD here [1].

And to save you a click, they compile it from a source tarball.

[1] https://git.archlinux.org/svntogit/packages.git/tree/trunk/P...

They even need the Google API key...

https://git.archlinux.org/svntogit/packages.git/tree/trunk/P...

They must already be doing a full build, test, and benchmark for each platform for each PR to their codebase in CI, presumably in parallel to make it run reasonably fast. It doesn’t seem like much of an extra step to release those binaries they’re already building hundreds of times a day.

We do exactly this for people who want. A reasonably stable build (including full testing, and benchmarking, which takes multiple hundreds hours, parallelized of course) is available twice a day (0:00 and 12:00 UTC). It self updates (either differential or full download depending of what landed on the past day).

I've been using only this for the last 10 years or so (on windows, Linux, Mac and Android), and I can count the number of time things really broke on one hand.

We call these Firefox Nighly, and they are available at https://www.mozilla.org/en-US/firefox/nightly/all/.

Release notes at available at https://www.mozilla.org/en-US/firefox/71.0a1/releasenotes/, but they are not always updated for each version, sometimes batched (its quite some work due to the amount of commits that are pushed per day).

There is a cool blog at https://blog.nightly.mozilla.org/ that talks a bit about the new stuff.

All this is really experimental and probably not for everybody, but helps us to deliver something good at the end. Filing bugs and giving us feedback is of course particularly appreciated, as always!

Are nightlies tested on CI for all platforms that Firefox supports ? I only see Tier-1 in that matrix (Linux, Windows, Mac). No iOS, Android, *BSDs, etc.

They already do that with Firefox Nightly. But most users expect a more stable product than that.

How can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?

To begin with, it is probably released with a subset of the CI matrix, because I imagine that the full CI matrix takes more than a day. Second, it is nearly impossible* to cover all possible environments+addons+websites and test them correctly. This isn't a normal desktop app, nor a webapp that always is running relatively sandboxed in a more or less known environment. Browsers are platforms, as OS are. Lots of people doing wacky stuff on it. Very hard to realistically automate all that stuff.

*impossible with a limited constraint of time and/or money.

It can be more stable by having nightly users test it beforehand for a while and report any new bugs.

What are those users doing that can’t be automated and run in CI?

It'd be pretty ambitious to have CI run against the variety of web apps that real users "test" against (including ones they're developing!)

I'm guessing something the devs haven't thought of. As soon as you hit 10k+ users, chances are people are doing things you didn't expect.

Test on exotic hardware, for example. Or exotic combinations of libraries in the OS which FF depends on.

Those should be in the CI test matrix then - we shouldn’t be relying on manual testing done specially for releases.

If even Microsoft can't catch these, why should Mozilla be able to? I am admittedly not in the CI business for frontends but I'd imagine that the combinatorics vastly outnumbers the possibilities of any organization.

I think it's more a matter of discipline than anything else. It's easy to get lost in the weeds with a ton of things waiting to be merged, and over and over saying "just one more PR to merge and then I'll release", and then it's a year later and you still haven't released.

Getting features out to people faster means shorter feedback cycles.

Regarding versioning, most applications don't use semantic versioning; that's mainly for libraries. Not sure what "backwards incompatible" would mean for a browser, anyway.

Addons, plug-ins, CSS and HTML directives, etc.

I would argue that you should do features detection rather than depending on version number, as much as possible.

Firefox probably had bad memories of when they "released when it's ready" during the FF3 -> FF4 days where a couple of features held back everything. With fix released cycles, they have more of a motivator to cut off things that aren't ready yet so that the things that are can go out.

"probably had bad memories"

Speaking as someone who was at Mozilla at that time, you can change "probably had" to "definitely has". In hindsight I'd go ahead and call it a disaster. The X weeks release cadence solves so many problems.

Remember back then when enterprises certified their entire install base against FF3, but then FF4 came out & they needed to redo all the enterprise certification bullshit.

zach_galifianakis_laugh.mp4

With shorter release cycles, there's also less pressure to ship half-baked features. If you know the next release is just four weeks away, then slipping a feature one release is rarely a huge deal. But if the next release is one quarter or one year away, teams rush to ship now and fix bugs in production later. Overall product quality will take a hit at every release.

Another thing that I observed in our culture since we switched is that it is easier to coordinate cross-team efforts - if I know X will be done in Fx70, I can plan my work on top of it for 71.

Such logic is commonly performed without releases until you start depending on multiple features in multiple places. Then it saves you.

The same "alignment" works also much better between engineering and non-engineering teams. Once we collect the focus areas and major features planned for release X, security, performance, accessibility, localization, platform integration, UI, UX, design, legal and marketing teams can prioritize their work to aid them.

Having no release cycle, or long release cycle, dilutes this effort and reduces visibility.

There is, in my opinion, somewhere, a threshold beyond which the release cycle burden outweighs the benefits and/or where the cycle is to short for teams to usefully react, but I don't think we hit it so far.

Totally. I think of this as the "next bus" effect. If you see your bus pulling up, do you run for it? If the next bus is in 5 minutes, no. If it's once a week, hell yes.

They're clearly moving gradually in this direction, but I think it's an open question how far they can get.

For server-side software, it's easy enough to do multiple releases per day, because you have full control over the environment, and all the work is internal. But for desktop/mobile software like this, having multiple releases per day won't be efficient in terms of bandwidth, load, or user experience.

And each browser version has other external costs, like documentation and support. Right now it can be hard to solve a problem like, "How do I do X with Firefox" because the answer varies by version. The more versions you have, the harder that is.

Or can you imagine trying to triage and verify bug reports if there are, say, 3 new Firefox releases per day, and people could be on any one of dozens of versions because they don't quit and restart the browser very often? We can imagine solutions to these problems, of course. But I think getting a release cadence below weekly for desktop apps will be a very long road.

This is a great overview of different release management strategies and the practical implications: https://www.youtube.com/watch?v=IKsQsxubuAA

Not part of the talk: if you have more than 3 people involved you probably need a release schedule in terms of either scope or time. I'd personally advise you to have one even if you only have 1 person, just for a mental target if nothing else.

The talk goes into studies of projects which did scope based releases and got horrible results in practice. All the ones in the talk switched to time based releases and were much happier. Also, in a software project, you can only fix either scope or time, but not both. Also, if you want to ship in a timely manner, you probably want to fix one of them, not let both float.

Edit: it looks like this is his paper on the topic: http://www.cyrius.com/publications/michlmayr_fitzgerald-time...

Probably for a similar reason to why c++ releases a new standard every 3 years. This blog post explains c++'s reasoning:

https://herbsutter.com/2019/07/13/draft-faq-why-does-the-c-s...

Good point, but factor in build/release, then deployment. We all know what some companies are like in their desktop real estate rollouts - if you have too frequent updates, you end up with greater disparity in releases with a template support of - oh that's not the latest, pls try again.

Then feedback, if you rollout release too frequently you also can be several releases in and get feedback of some bug in a release prior to current release, albeit only a week or so. So by having organised releases, allows better management and a better experience overall. It's a fine balance, but for those who needs it, there will always be nightly builds of most things they want with last second updates, also self-builds for those inclined.

But I get the whole artificial timeframe does seem somewhat odd, but the further you look at the overall usage, management and support factors et all, you start to appreciate such regular timeframes.

On the other extreme, I've worked in an environment in which a whole suit had to be retested and go thru all the hopes as a full stop was missed of a comment and that change caused the whole testing, management/release cycle to go thru the motions again. That you would like, ignore and package up with other small things into a full release - which is what many do these days in most environments. After all - would you want to go thru a rollout of a new release that may contain a fix you want, or equally may just be adding a full stop to a comment in the code and yielding no binary level changes, albeit that it went thru the whole hoops and hurdles as a full release and that is what you got.

So from my experience, a 4 week release cycle is good, fixed, something that you can plan around. Also kudos for not just picking a day a month and doing monthly updates like most, that tends to seem more arbitrary unlike 4 weeks, which has an air of less arbitrary and feels like it was thought out more. Though an element of personal experience does come into play.

> kudos for not just picking a day a month and doing monthly updates like most, that tends to seem more arbitrary unlike 4 weeks

Planning feature development and rollout is more difficult if you always have to refer to a calendar to find future release dates. I would prefer to be able to say a feature can land in Firefox Nightly in January, ship to Firefox Beta on February 1, and Release channel users on March 1.

Mozilla has a long wiki page of past and future release dates:

https://wiki.mozilla.org/RapidRelease/Calendar

The problem there is February 1st isn't always a good release day, say if it's a Saturday. So you'd ship on the 2nd or 3rd sometimes... or sometimes that's a holiday, or there are validate reasons to avoid a period (like the first ~week of July or September). You could try to pick something like the "2nd Wednesday" of the month, but there are still occasions for mid-week holidays and other timing problems related to marketing or external events (other announcements, or world events that would clobber your marketing push). Inevitably you have to plan it anyway, so you might as well do that from the start.

For quite a few products, releasing takes time. QA, auditing the security, localizing, generating the builds, executing exhaustive in-depth unit tests, flushing caches, writing release notes, publicating on different websites... it can take a while!

When the cost of the release goes above a certain amount, due the sheer number of features or the thoroughness of the release process, it starts to make sense to mutualize release tasks between a bunch of features.

Bundling those features on a time basis is quite often a local optimum between an acceptable delay for the product team and an acceptable cost for ops/release/qa/...

> Why release on a schedule at all?

I feel like it might help with motivation. It's easy to push things "to tomorrow" if you don't have a hard cap on how many tomorrows are available before the feature is supposed to go out.

> If people demand a certain new feature ("[this will] bring you new features more quickly"), it can just be brought out as it's ready, instead of having to wait an average of 3 (now 2) weeks before it can be released?

It's kind of like forming a habit. Having a set schedule could in theory delay quick features, but it will also keep the motivation going for time-consuming features.

> And why does this have to increment the major version, since when is every single update backwards incompatible?

I actually really like this personally. Encoding compatibility in a version number is, IMO, useless— it's sometimes just hard to know if your changes will break something or not, so there's a wealth of minor versions out there that break things anyway.

Just using a single number just makes everything simpler. Detecting breaking changes should be done with systems (tests, static analysis, etc) and not by trusting a number some developer came up with.

One issue with releasing every day or every hour is that that would mean users downloading massive numbers of updates. Sure, you ship deltas, but there's still a cost to each update.

At my work the culture is just to ship changes as soon as they improve the product/codebase

Doing that for a binary release that is in dozens of platforms, configurations, locales, etc. is incredibly time consuming. Uptake on binaries is also usually on the order of days-weeks before most of your user base is up to date. It's easier for say a website or service that can deploy to millions of users within minutes.

Yeah, I agree for sure. I get tons of updates for spotify, discord, and other applications that don't need to do complex binary distribution

>why release on a schedule at all

Exactly my thought - why 4 weeks? Why not just put everyone on Nightly and be done with it? Then everyone will enjoy the firehose of agility and features that is Mozilla! You're always shipping working code, right? If it passes the harness, it's good, isn't it?