Apparently I missed `app.normandy.enabled`, because I think I would've remembered a name with connotations of a bloody massive surprise attack.
Incidentally, `app.normandy.enabled` defaults to `true` in the `firefox-esr` Debian Stable package. Which seems wrong for an ESR.
For personal use (not development), I run 3 browsers (for features/configurations and an extra bit of compartmentalization): Tor Browser for most things, Firefox ESR with privacy tweaks for the small number of things that require login, and Chromium without much privacy tweaks for the rare occasion that a crucial site refuses to work with my TB or FF setup.
Today's crucial cert administration oops, plus learning of yet another very questionable remote capability/vector, plus the questionable preferences-changing being enabled even for ESR... is making me even less comfortable with the Web browser standards "big moat" barrier to entry situation.
I know Mozilla has some very forthright people, but I'd really like to see a conspicuous and pervasive focus on privacy&security, throughout the organization, which, at this point, would shake up a lot of things. Then, with the high ground established unambiguously, I'd like to see actively reversing some of the past surveillance&brochure tendencies in some standards. And also see some more creative approaches to what a browser can be, despite a hostile and exploitive environment. Or maybe Brave turns out to be a better vehicle for that, but I still want to believe in Mozilla.
Edit: There are some questions about whether Normandy is really enabled in Debian Firefox ESR even if the about:config setting defaults to true. I've filed a bug report, and I'm sure once a Debian maintainer has a chance to look at it we'll find out the answer.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928433
Edit2: It should go without saying, but please do not spam this bug report with "me too" and its ilk.
?
>:-(
Grrr.
I'm just getting old and curmudgeonly maybe? I've decided though, I'm starting an animated security blog to show people the ludicrousness of all this kind of stuff in plain language. I'll be Statler, and I just need someone to be Waldorf. Because this stuff really is getting Statler and Waldorf level ridiculous.
You're not. You just have standards.
We need people with standards in this industry, because that's the only source we have of market signals that prevent the market from going full user-hostile.
That's needless drama. They will be rolling out the fix in a point release. Whatever way you use to update your browser will install that and get the fix. So the worst case is just going back to the old days where you'd have the issue until your distro issued a new package or you manually updated the browser version on Windows or OSX. What exactly would you expect that's not exactly what's happening?
That's the way it always works, isn't it? Security and convenience are opposing concerns.
The three goals of computer security are integrity, confidentiality and availability.
All three of those expand the usefulness of the system to the end user.
A configuration where Mozilla cannot push remote updates is neither more secure nor less secure. Mozilla is often under fire for not allowing a privacy conscious, minimal trust use case.
Clearly more eyes are good, but... In between “Wild West WebExtensions” and “Mozilla backdoors my Firefox and it gets used for nefarious purposes” and “delays in browser updates increase exploitation windows”, I know which threat models I’m buying.
Then, even if developers keys and computers are compromised, I would notice something is wrong.
* No, of course that I don't always do that. I even don't often do that. But I did do that in the past, and I'd like to have the option to do that.
Why even have an official channel, providing visibility and official oversight, if when it comes down to it, you're just gonna push remote code updates through the same side channel a potential hacker would use?
People are saying it's for convenience. OK, but then they have to understand that doing things in that fashion is a really bad look. And now your users are set up to believe that, at least some of the updates coming from the side channel are "trust"-able.
Reportedly, Firefox only checks the date once per day, so if it hasn’t yet checked for you today, this will be the result.
> I looked in about:config and lo and behold, app.normandy.enabled=default [true].
I would assume that the config setting only has any effect if the feature is available in the build. Which it isn’t in Debian.
Has Mozilla provided instructions to manually fix the issue? if so where? (XORcat was helpful to provide a solution, but I refuse to apply it if it doesn't come from Mozilla itself...)
In the Debian bug about this issue¹ it says “Firefox from the Debian package has data reporting disabled so using studies is not possible.”
FYI, I have learned from other user's comments and the Wiki page below that Studies and Normandy are different things. The former depends on the latter, but not vice versa. So it is possible that Debian disabled the studies program but did not disable the underlying Normandy tool. You might also want to look at whether firefox is affected in addition to firefox-esr.
https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout#...
Weird.
I've closed the above bug report as it's not really a bug.
Unchecking "Allow Firefox to install and run studies" in the UI does not change "app.normandy.enabled" to "false".
Then, does unchecking "Allow Firefox to install and run studies" really disable Normandy, or not?
> Preference rollout is meant for permanent changes that we are sure of. Shield is meant for testing variations and figuring out what, if anything, is the best thing to do.
https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout#...
And if you look at the big normandy JSON, hey, it's all the same Pocket and heartbeat shit we've seen from studies.
One can guess based on the wiki page that the answer is "no", but that's just a guess.
>12:50 p.m. UTC / 03:50 a.m. PDT: We rolled-out a fix for release, beta and nightly users. The fix will be automatically applied in the background within the next few hours, you don’t need to take active steps.
>In order to be able to provide this fix on short notice, we are using the Studies system. You can check if you have studies enabled by going to Firefox Preferences -> Privacy & Security -> Allow Firefox to install and run studies.
>You can disable studies again after your add-ons have been re-enabled.
>We are working on a general fix that doesn’t need to rely on this and will keep you updated.
I refuse to enable studies, even temporarily. This comes very close after the IE6 conspiracy revelation, where ends justifies the means.
Please provide a link to the certificate file, and step by step instructions for installing it, without enabling and conflating with mozilla studies...
hotfix-update-xpi-signing-intermediate-bug-1548973: https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...
From the looks, it installs the above plugin, and changes `app.update.lastUpdateTime.xpi-signature-verification` to `1556945257`
I can't get it to work in ESR 60 though. Getting file not found on "resource://gre/modules/addons/XPIDatabase.jsm"
edit: The linked XPI definitely seems to add the new certificate, whatever mechanism used to reverify the signatures just doesn't seem to work in 60.
edit2: Restarting Firefox appears to have forced the reverify... Possibly a flag that I twiddled with though, hard to be sure. Either way, the above should help people get everything running again without having to enable studies/normandy.
The hotfix extension does two things:
1) Install a new certificate for "CN=signingca1.addons.mozilla.org/[email protected]", effectively replacing the old certificate that expired. This should work.
2) Then it tries to import the internal "resource://gre/modules/addons/XPIDatabase.jsm" module and calls XPIDatabase.verifySignatures().
This does not work on ESR, as "XPIDatabase.jsm" is a new-ish thing that isn't present in ESR yet. In ESR the function is still in "resource://gre/modules/addons/XPIProvider.jsm" (XPIProvider.verifySignatures()). Thankfully, the non-existing module is imported using ChromeUtils.defineModuleGetter, which only lazily loads the module on first of the imported property, so after the certificate-adding code has run.
https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...
Can mozilla please verify, confirm authenticity, and list this instruction on their issue page?
I encourage you to go through the whole Normandy process yourself in a test environment, and even better (if possible), check out the code to see whether it looks legit or benign.
I'm happy, because I went through and checked it out myself without needing to enable Normandy on my actual Firefox, but ultimately, it will be great when Moz can get instructions for manually applying the fix out.
This fixed it for me. Thanks. W10/FF 66.0.3
(I would have wanted to read my comment if someone else had written it, so by the golden rule I make the comment I wish I had read)
What?!
Yes, Youtube put up a banner asking IE6 users to move to a more modern browser 10 years ago. How is that in any way related to Firefox pushing a hotfix in 2019 to fix a certificate issue? Are you worried there is a big evil conspiracy to use this mechanism to uninstall Internet Explorer from peoples' computers?!
Firefox, it turns out, has a built-in telemetry system that defaults to enable exactly the same behavior: changing your system, to suit their desires.
You’re words “a big evil conspiracy to use this mechanism to uninstall Internet Explorer from peoples' computer” are misleading. No one would propose that the intent is an attack on Microsoft applications. Rather, the intent is to blindfold users on a whim, should a Firefox component prove inconvenient to the providers of Firefox. Ostensibly, in the event that some add-on or extension threatens the bottom line for major backers of Firefox’s funding.
An example of the typical use of this system: say Mozilla wants to enable video hardware acceleration in Firefox but they don't know if bugs in video drivers or in Firefox will make crashing more frequent. So they enable hardware acceleration for 1% of users instead of 100% and compare the reported crash rate between the two to determine if it's ready to be pushed out universally.
You say it is "typically" used for benevolent purposes, but why should we trust Mozilla? Mozilla does not have a stellar history with this sort of thing and in my experience they do not take security as seriously as they should if we are to trust them with such a feature.
Mozilla has had several "PR nightmare" decisions that a vocal set of users didn't like, and sometimes were genuinely ill advised/bad/shitty. But as far as I can see they do not have a bad track record when it comes to security/privacy. Do you have any examples of actual serious security/privacy fuck ups by Mozilla/Firefox? I mean that stood up to scrutiny beyond the sensationalist headlines?
Their defaults might not be your defaults, but they are even working on bringing Tor into mainstream Firefox. None of this means they are above criticism of course, but... context!
The sum total of their actions points towards an organisation that has some internal problems but that is genuinely pursuing privacy and an open web as a goal for as many users as possible.
I mean, they are currently shipping real actual ads on the new tab page that aren't blocked by ad blockers - and possibly can't be (there are limits to what WebExtensions can modify on Firefox internal pages). Sure, maybe your parent comment was exaggerating a little bit, but what if Mozilla instead starts inserting "privacy-friendly" "recommendations" into webpages in order to "enhance users' browsing experiences"? That doesn't sound at all far-fetched for the Mozilla we know today.
This is exactly the sensationalist misrepresentation I was talking about. You don't like what they are doing, fine. Misrepresenting it as something that it's not is not fine.
Besides: Mozilla is funded in large parts by having Google as the default search provider. This means they are funded by Google selling ads. Them starting up new revenue streams and getting away from that funding model would be a pro privacy step.
[1] If you are referring to something else that I missed, feel free to enlighten me.
See: https://help.getpocket.com/article/1142-firefox-new-tab-reco... especially the part that says "From time to time, the occasional sponsored story may appear as a recommendation from Pocket. These stories will always be clearly marked, and you have control over whether they’re shown on your new tab page."
All so-called recommendations I've seen have been spammy, the sort of stuff you see linked as "other articles you may enjoy" when you disable your ad blocker on bad sites. Regardless, this directly contradicts your claim that there haven't been incidents of sponsored content on the new tab page: this is explicitly what is happening according to Pocket's own website. Mozilla themselves explicitly said they are introducing sponsored stories to the new tab page: https://blog.mozilla.org/futurereleases/2018/01/24/update-on...
I think there's a world of difference between making a search engine that sells ads the default, and selling ads yourself and inserting them into the browser's chrome. Among other issues, if I help someone install an ad blocker, that ad blocker will block ads on Google, but will not block ads in the browser chrome.
So, given this and other recent behavior by Mozilla, I have to say I don't think seeing "related stories" inserted into the browser chrome for certain web pages is at all far fetched. That should worry us.
I actually don't see the pocket recommendations on my desktop (maybe the Linux Mint build has them disabled by default), but they are there on mobile. There is a UI setting to disable them of course. It's explained right on the page that you link to.
More importantly, that page also explains that no data gets sent to Mozilla or pocket or anyone else for these ads to show up.
So again, no privacy violation here. I also think it's an extreme leap from "they show this in the new tab page which they design and control" to "they could start showing it overlayed on other peoples content".
I think they got some decisions very wrong. Among them not implementing a way to allow people to override signing of addons, which people did warn about. Having signatures enforced as a strong default is certainly good and right, but if they had included a "right click on addon, use without signature (WARNING THIS IS SKETCHY REAL ADDONS DON'T ASK YOU TO DO THIS)" option this signing issue would have been relatively mild.
But their track record on privacy/security simply isn't as bad as people make it out to be.
Sadly I don't, but others argue they have top notch standard security practices like automated alerts etc. regarding certificate renewals...
This one isn’t very privacy-friendly or open. And that raises all the previous questions again. Should they maybe have learned something about clandestinely fucking with people’s systems?
They added a dismissable banner. That falls far short of "changing their experience", in my mind.
Is the existence of a back door method of updating Firefox preferences something that will be disclosed to users? What about a UI knob to disable it?
Apparently, there is no one associated with browsers can be trusted in the least.
[1]: https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout
This is the first I have heard of Firefox changing my config settings invisibly in the background. This is obscene. Who on earth thought this was a good idea? The security ramifications are limitless.
I understand all too well that most companies have decided to start A/B testing things on subsets of users, but that doesn't mean you should force that mode of thinking into everything. What a horrible decision. I don't recall ever seeing any news or notifications or checkboxes about studies or "Normandy" at any point.
Are there some other good open source alternatives to Firefox? I remember hearing about Brave but also that it was tied into some cryptocoin nonsense, so I'm not sure what else to look at.
you must not have been paying attention the last 3 or so years
Mozilla is doing all kinds of, IMO, unethical things with FireFox that goes against the core value of the mission statement of the Mozilla Foundation.
They are too busy trying to replicate Chrome to care about privacy, security, or basic user rights
I didn't realize what a true mess Mozilla had become.
Looking Glass, Pocket, Banning Plugins based on Political ideology, Backdoors like Normandy, and the STUDIES system, their creation of what amounts to Mozilla version of the Ministry of Truth, Their partnership with Cloudflare to send everyone's DNS to Cloudfare over HTTP, and whole host of other things
I've been using brave because of that: all of that is baked in so my only extension is my password manager
It will even be documented for them: https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout
> What about a UI knob to disable it?
app.normandy.enabled
That is not what I meant by a UI knob, and I sure hope you knew that. By UI knob I mean something easily discoverable and self-explanatory. Rooting around a gated (with a mighty strong warning, I should add) config section for something called "normandy" is not intuitive, and it's not self-explanatory.
And I sure hope that by disclosed to users I did not mean some Hitchhiker's Guide-esque disclaimer on a wiki page. Something as (potentially) insidious as a preferences backdoor should absolutely be disclosed to users with the same level of visibility as the stories nonsense.
Perhaps "normandy" is entirely harmless, but you guys lost a metric fuckton of credibility by using your backdoors to spam people[1]. Playing coy does nothing to improve your credibility or reputation.
1: https://www.theregister.co.uk/2017/12/18/mozilla_mr_robot_fi...
Options -> Privacy & Security > Allow Firefox to install and run studies
Edit: Source:
See: https://discourse.mozilla.org/t/certificate-issue-causing-ad...
> In order to be able to provide this fix on short notice, we are using the Studies system. You can check if you have studies enabled by going to Firefox Preferences -> Privacy & Security -> Allow Firefox to install and run studies.
Normandy seems to be the internal name for this system: https://github.com/mozilla/normandy
Edit: rephrase for clarity
I happen to be one of the users with Normandy disabled, so I'm foobar'd anyway. That said, the reason I disabled it is because it is a security hole you could drive a semi-truck through. And now they want us to enable it to provide a "fix" for the secure way in?
I thought I was the only one who saw a problem with that. Your post is evidence that I'm not completely off in my thinking.
But there are trustworthy people working with and integrating that code, there's a good chance they'll notice a hinky commit, and they're very close to having completely reproducible builds—which means that there can be verification that the shipped binary matches the inspected source.
https://gregoryszorc.com/blog/2018/06/20/deterministic-firef...
I guess "easier" isn't the word really, because Chrome can't really ever be locked down. It's pretty much always, effectively, an open book to Google.
You can lock down everything in Firefox. The drawback being, of course, times like this, when you can't get the fix unless you leave Normandy enabled. (Which I didn't.)
>:-(
Grrrr.
Where are you getting this from? AFAIK all Mozilla code / prefs they can push should be signed -- this very issue seems to stem from the cert used to sign AMO extensions expired.
Well it's a half-assed knob then, because it was unchecked and still I had app.normandy.enabled = true somehow.
I agree Mozilla approach to stuff like this is... less than ideal.
Where are you getting this from?
TL;DR you are not sending us your browsing history.
If telemetry and studies were turned off, your browser wasn't sending us this unique id.
If you had kept them enabled, for normandy telemetry you would have been sending us the data described at https://firefox-source-docs.mozilla.org/toolkit/components/n...
You can read more broadly about what data Firefox sends by default at https://www.mozilla.org/en-US/privacy/firefox/
And learn more about the review process any data collection has to go through at https://wiki.mozilla.org/Firefox/Data_Collection
> It will even be documented for them:
That sounds like you do not think the concern is warranted. I've used Firefox since the first time it was available, and Netscape starting with the first ever betas. At no point was there a dialog that said "Do you want us to be able to change your browser settings remotely?"
>> What about a UI knob to disable it?
> app.normandy.enabled
That is not a "UI knob" by any stretch of the imagination. Looking in about:config revealed:
app.normandy.logging.level
Is there a way to find out what is being logged and why?
So, the question can be rephrased as "is the fact that Firefox has been logging all users' entire browsing history despite the fact that the user has not chosen to set up a Firefox account going to be disclosed?"
Chill out, this preference only determines what is logged locally (never sent to the server). It's a debugging tool.
Sources: - https://searchfox.org/mozilla-central/source/toolkit/compone... - https://searchfox.org/mozilla-central/source/services/common...
1. being completely transparent about all the mechanisms that data or code can be pushed to or pulled by the browser, or pushed from or pulled from the browser; and
2. having a toggle for all of them, yes every single one, in Privacy & Security.
> Normandy Pref Rollout is a feature that allows Mozilla to change the default value of a preference for a targeted set of users, without deploying an update to Firefox.
Rolling out a new certificate goes beyond changing the default value of a preference which rightly raises questions about what else Normandy allows which is not documented.
I fail to see both the "UI" and the "knob" part of this. Why is this not a checkbox in the preferences? Why do so many people not know about Normandy?
This is such a gigantic mess, even for a very loyal Firefox user it's to swallow.
I was browsing fine this morning for maybe four hours and now all my plugins/addons are gone.
Edit: I had to click "Restart with addons disabled (safe mode)" for those wondering.
What do you mean?
It's on the level of saving passwords as texts in terms of privacy.
uBlock still reads your bank website, etc.
Also here's the code for the server: https://github.com/mozilla/normandy
Mozilla should follow up with a post describing exactly how Normandy works and the full capabilities it gives them.
I think people here can be expected to read some code if they are interested in how something works.
I expect code related to normandy to be ~1k LOC in size and probably written in JS. I haven't checked though, because I don't really care today.
Got it.
Oh well, at least we don't have another season of Mr Robot spam to look forward to.
if (studies.enabled) {
if (normandy.enabled) {
...
}
}???
It's a publicly documented feature with a publicly documented way to disable it.
Like, "In order to disable Normandy, uncheck Vaduz, Monterrey, Vologda and select Newcastle in the Xinjang drop-down menu - we'll ship Bronx with next update".
P.S. Keep flagging, I'll repost, no problem.