News for hackers. Stuff that matters.
Re:Technical Details on the Recent Firefox Add-On Outage by jzl on Thursday May 09, 2019 @05:00PM (#19872374)
Looks like a good read. I haven't finished reading it yet, but there's something I still don't get ...

Windows and macOS both have a signing infrastructure for apps. The rules of that infrastructure dictate only that apps must have been signed by a valid certificate at the time they were signed. That way old app downloads don't need to be periodically re-signed just to account for expiring certificates. I can download a 5-year-old version of 7zip or whatever and it runs just fine because it was signed with something valid to the timestamp in the signature. The process of distributing desktop apps would be utterly insane if this were not the case.

Not following this model for browser plugins seems unnecessarily cumbersome. Is it really worth requiring all browser plugins to be signed by a currently valid certificate? Is there a document or blog post where this is argued to be more appropriate?

I get that it arguably leads to more stringent security, but I'm not convinced by the delta improvement of that model over the desktop model, given the additional downsides. And the "let everything expire after a few years and resign it" process should not be used as a substitution for revocation. After all, if it were determined retroactively that a malware extension was signed does it really help everyone that it can't be loaded in a year or two, given the damage it could cause right now?

Re:Technical Details on the Recent Firefox Add-On Outage by johnp_ on Thursday May 09, 2019 @05:45PM (#19872745)
It seems Mozilla is in the process[0] of moving the signature scheme to COSE, which allows timestamping[1]. A code comment[2] says that the current package format doesn't allow it.

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1545836

[1] https://tools.ietf.org/html/rfc8152#section-4.5

[2] https://searchfox.org/mozilla-central/rev/b9da45f63cb5672449...

Re:Technical Details on the Recent Firefox Add-On Outage by josteink on Friday May 10, 2019 @02:05AM (#19875443)
If that is true, they should mention this in the post-mortem.

Code signing is a well understood problem with a well known solution, but the blog post discusses everything except the well known solution.

Right now you have a problem caused directly by lack of time stamping, and the article doesn’t even acknowledge that.

That’s not inspiring confidence. I’m genuinely still not sure if they have understood what the actual problem is and how to solve it properly.

Re:Technical Details on the Recent Firefox Add-On Outage by Vinnl on Friday May 10, 2019 @05:19AM (#19876266)
> If that is true, they should mention this in the post-mortem.

They might:

> We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make

The lessons noted down here are just some thoughts by the author of this blog post:

> but in the meantime here are my initial thoughts about what we need to do.

Re:Technical Details on the Recent Firefox Add-On Outage by hunter2_ on Friday May 10, 2019 @03:18AM (#19875767)
Exactly, the article specifically calls for inventorying, not eliminating, "ticking time bombs." As for the inventory, dealing with non-ACME certificate creation without some kind of calendar/reminder mechanism is pretty crazy to me.
Re:Technical Details on the Recent Firefox Add-On Outage by tialaramex on Friday May 10, 2019 @03:32AM (#19875844)
Nice password by the way.

I thought the mention of "ticking time bombs" showed someone is thinking about this properly because end users get the same experience if e.g. a timer gets treated as negative in 2038 or the browser depends on the century field being 20 as they do if an X.509 certificate expires. If you are sure you handled all certs, but you blow up because your GPS epoch wrapped then you still screwed up.

Re:Technical Details on the Recent Firefox Add-On Outage by rst on Thursday May 09, 2019 @05:20PM (#19872548)
I share the confusion. I can understand wanting to re-check addon validity somehow to allow for recalls on malicious addons that somehow slipped through. But I'm not sure what threat is mitigated by allowing a recheck to fail just because the clock has advanced, if all the certs were valid at the time of the installation. (Unless placing time bombs is the actual intent?)

The post did say that they'll "be looking more generally at our add-on security architecture to make sure that it’s enforcing the right security properties at the least risk of breakage". I hope they'll be looking at that point here; if they want to support explicit revocation, there may be other ways to do it which aren't quite so prone to invocation by accident (e.g., publishing a revocation list signed by certs valid at the time of revocation).

Re:Technical Details on the Recent Firefox Add-On Outage by ddalex on Friday May 10, 2019 @04:28AM (#19876079)
The idea here may be to invalidate certain add-ons after they have been released in the wild by revoking the certificates used to sign them.
Re:Technical Details on the Recent Firefox Add-On Outage by acqq on Friday May 10, 2019 @06:35AM (#19876569)
Only one certificate was used for all add-ons, its expiration disabled all signed. No “certain add-ons” it was “all.”
Re:Technical Details on the Recent Firefox Add-On Outage by jlgaddis on Friday May 10, 2019 @12:32PM (#19879396)
Per the article, there is a separate "end-entity" certificate for each add-on. They were all signed by the same "intermediate" CA, however.
Re:Technical Details on the Recent Firefox Add-On Outage by acqq on Friday May 10, 2019 @03:39PM (#19881508)
> They were all signed by the same "intermediate" CA, however.

And the expiration of that "intermediate" obviously should have not made already accepted add-ons stop working, in this specific use-case. It's not about establishing a new trusted communication channel for a new content, it's not about disabling some specific add-on.

Thinking logically, that was not the role of that immediate certificate.

So the behavior was clearly designed wrong, because completely wrong analogy was used -- that of creating a connection, where expired intermediate certificate should prevent the new connection, which could an transfer a new attack if not verified. Here the verification already happened, and the "ban" of a specific set of add-ons was also clearly not a case.

Additionally it seems the handling of the update of the expired intermediate was not the topic of the design at all.

Re:Technical Details on the Recent Firefox Add-On Outage by johncolanduoni on Thursday May 09, 2019 @05:14PM (#19872494)
For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same.

I’m not saying they shouldn’t, but it is a significant piece of complexity.

Re:Technical Details on the Recent Firefox Add-On Outage by reificator on Thursday May 09, 2019 @05:18PM (#19872531)
> Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same.

Mozilla already requires uploading even your private extensions to a Mozilla server to be signed for internal or external deployment.

https://wiki.mozilla.org/Add-ons/Extension_Signing

Re:Technical Details on the Recent Firefox Add-On Outage by Boulth on Friday May 10, 2019 @03:30AM (#19875836)
> Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same.

Nope. Actually all CAs that offer code signing certificates are required to provide timestamping service compatible with RFC 3161. These timestamping servers are usually free to use, see for example https://knowledge.digicert.com/generalinformation/INFO4231.h...

Re:Technical Details on the Recent Firefox Add-On Outage by eps on Thursday May 09, 2019 @05:54PM (#19872810)
Every major CA runs a trusted timestamping service. Mozilla doesn't need to maintain their own timestamping infrastructure, they could delegate to one of the CAs, probably based on some sort of formal agreement with them.

Though the way things stand, all CAs have no problem timestamping sigs made with certs that are from other CAs, so perhaps even no explicit agreement is required.

Re:Technical Details on the Recent Firefox Add-On Outage by temac on Thursday May 09, 2019 @07:24PM (#19873507)
The checker just has to have roots tracing to timestamping CAs.

I really don't understand why Mozilla designed their system like they did. Code signing is well known and probably even done for the Windows installer of Firefox, why did they just not duplicate the model? Checking expiration vs current time makes absolutely no sense for code signing, esp. at runtime (it could kind-of sort-of make a little bit of sense at install time, but I'm not really convinced, and maybe it is even not really possible to distinguish between the two with an effective boundary)

Re:Technical Details on the Recent Firefox Add-On Outage by StavrosK on Thursday May 09, 2019 @06:27PM (#19873056)
Can you explain why you need the current time anywhere in this? Say I download a ten-year-old addon, it's signed by a valid signature from that root, with a valid signing date. What's the problem?

Are we worried that someone will steal an old/expired cert and have control over a user's clock?

Re:Technical Details on the Recent Firefox Add-On Outage by cortesoft on Thursday May 09, 2019 @09:22PM (#19874239)
Imagine you have an old expired key... you take your new malicious extension, and sign it with the expired key and a time stamp that says it was signed at a time the key was still valid.

Without some other verification mechanism, you can't tell the difference between this and an actual signature signed when the key WAS valid

Re:Technical Details on the Recent Firefox Add-On Outage by josteink on Friday May 10, 2019 @01:30AM (#19875309)
> sign it with the expired key and a time stamp that says it was signed at a time the key was still valid.

The whole point of a trusted timestamp is that such signature cannot be made for a fraudulent date, otherwise it would be utterly pointless.

This scenario and threat model does not exist if timestamping is correctly implemented.

Re:Technical Details on the Recent Firefox Add-On Outage by tialaramex on Friday May 10, 2019 @04:00AM (#19875971)
Nope.

RFC 3161 timestamps - which is what we're discussing here - can be fraudulently constructed for any timestamp value by someone who has the private key for the TSA (Time stamping authority). So what your parent described is easily possible: A system that relies on RFC 3161 timestamps has to trust that

* any cryptographic hash algorithms used remain safe

* any public key signature methods used remain safe

* the TSAs retain control over their private keys for as long as you continue to accept timestamps from that TSA

This is a big ask, and in practice the code signing systems you're probably thinking of just don't care very much. A state actor (e.g. the NSA) could almost certainly fake materials for these systems, we know that this has been done (presumably by the NSA or Mossad) in order to interfere with the Iranian nuclear weapons programme in the past.

You _can_ build a system that has tamper evident timestamping, but it's much more sophisticated and has much higher technical requirements. That's what drives the Certificate Transparency system. CT logs can prove they logged a specific certificate within a 24-hour period, and monitors verify that their proofs remain consistent, the to-be-built Gossip layers allow monitors to compare what they see in order to achieve confidence that logs don't tell different stories to different monitors. But to achieve this a CT log must be immediately distrusted if it falls off line for just 24 hours or if an error causes it to not log even a single timestamp certificate it issued. Massive Earthquake hit your secure data centre and destroyed the site? You have 24 hours to get everything back on line or be distrusted permanently. Bug in a Redis configuration lost one cert out of 4 million issued? You are distrusted permanently. Most attempts to build a CT log fail the first time, some outfits give up after a couple of tries and just accept they're not up to the task.

Re:Technical Details on the Recent Firefox Add-On Outage by stupidthrottle on Friday May 10, 2019 @04:23AM (#19876061)
> can be fraudulently constructed for any timestamp value by someone who has the private key for the TSA

Sure. Which is why these are heavily secured and guarded. Just like the keys for any cert, and highly trusted root certs in particular.

Any private/public crypto system can be compromised if the private keys are leaked. Everyone knows that.

That however is in no way a good argument for not using timestamps.

Re:Technical Details on the Recent Firefox Add-On Outage by tialaramex on Friday May 10, 2019 @04:58AM (#19876198)
RFC 3161 timestamps are used because they let people do something Mozilla doesn't care about at all and which was largely irrelevant here.

Alice the OS Vendor wants to let Bob the Developer make certificates saying these are his Programs, she is worried Bob will screw up so his cert needs to have a short lifetime, but her OS needs to be able to accept the certs after that lifetime expires so users can still run their Programs. So, Bob makes certificates and uses Trent's public TSA that Alice authorised to prove they were made when they say they were. Alice only has to trust Trent (who is good at his job) for a long period, and Bob who can be expected to screw up gets only short-lived certificates.

But Mozilla's setup doesn't have these extra parties. There is intentionally no Bob in Mozilla's version of the story, they sign add-ons themselves, so timestamping plays no role. If a 25 year TSA would be appropriate (hint: it would not) then a 25 year intermediate cert would be just as appropriate and simpler to implement for Mozilla.

Re:Technical Details on the Recent Firefox Add-On Outage by StavrosK on Friday May 10, 2019 @06:37AM (#19876578)
Oh, the developers are signing their own extensions? I wasn't aware how it worked and I was missing that part, I thought Mozilla signed them on upload (and thus could trust itself to not be malicious).
Re:Technical Details on the Recent Firefox Add-On Outage by cortesoft on Saturday May 11, 2019 @09:19PM (#19889488)
Well, Mozilla does the signing, but we only know that to be true as long as they control the private key. The whole point of expirations is to make mitigate the risk of an older key being stolen (or cracked).

So yes, Mozilla signs the extensions, but that doesn't change the importance of keeping the private key private... that is HOW we know it is Mozilla doing the signing

Re:Technical Details on the Recent Firefox Add-On Outage by jldugger on Thursday May 09, 2019 @10:55PM (#19874656)
How did you get ahold of an expired key?
Re:Technical Details on the Recent Firefox Add-On Outage by SamReidHughes on Thursday May 09, 2019 @11:27PM (#19874813)
The whole point of key expiration is that somebody might get a hold of it (or crack it).
Re:Technical Details on the Recent Firefox Add-On Outage by johncolanduoni on Thursday May 09, 2019 @07:14PM (#19873424)
Not current time, but time of signing of the executable (not signing of the code sign certificate itself). If you (as the OP suggested) use time of signing instead of current time, the whole point is you’re not using the user’s clock anymore.
Re:Technical Details on the Recent Firefox Add-On Outage by SAI_Peregrinus on Thursday May 09, 2019 @07:42PM (#19873617)
Mozilla is already packaging the executable inside an archive. That archive contains the executable, manifest, and any other needed files. The signer can add a timestamp, either as a new file or as a part of the manifest. Then the timestamp is signed along with everything else, and can be checked for validity.
Re:Technical Details on the Recent Firefox Add-On Outage by johncolanduoni on Thursday May 09, 2019 @07:57PM (#19873730)
If all extensions are signed by Mozilla’s own certificate, then adding a timestamp won’t do anything for the reason I outlined above and they should just turn off expiration validation altogether. The time of code signing check is only useful if you want to trust one certificate (the developers) to sign things only within the duration noted in the cert, but are willing to trust another certificate (e.g. Microsoft’s timestamp cert) forever. If there’s only one certificate in play owned by the authority for the whole system, then there’s no point in the timestamp at all (for validation purposes).
Re:Technical Details on the Recent Firefox Add-On Outage by Dylan16807 on Friday May 10, 2019 @05:04AM (#19876222)
It depends on the level of security you want.

Let's say you compromise by letting old signatures stay valid, but only for a year. And you rotate the intermediate cert every 90 days.

This system is more secure than the old one, because you only have to worry about key leaks for 15 months, instead of years.

But at the same time, it's impossible to have this giant wave of everything failing at once. Instead the best case is nobody can sign extensions for a couple days, and the worst case is extensions that updated exactly a year ago start to fail in real time.

Re:Technical Details on the Recent Firefox Add-On Outage by jzl on Thursday May 09, 2019 @05:20PM (#19872549)
Ahh, right. Good point. I wonder then if the current system was a deliberate choice over the timestamp method, or just deemed easier to deal with on their end.
Re:Technical Details on the Recent Firefox Add-On Outage by contravariant on Thursday May 09, 2019 @06:11PM (#19872952)
Maybe I'm jumping the gun here, but doesn't is seem that this kind of problem is exactly the kind of problem where a blockchain solution would be useful?

Or at least more useful than the 'when all you have is a hammer' style applications of blockchains.

Re:Technical Details on the Recent Firefox Add-On Outage by simias on Thursday May 09, 2019 @06:27PM (#19873052)
Given that you have a clear authority here (Mozilla) I don't see why you'd query a blockchain instead of querying signatures.mozilla.org

As always Blockchains are a good solution only for a tiny group of problems.

Re:Technical Details on the Recent Firefox Add-On Outage by semi-extrinsic on Friday May 10, 2019 @12:49AM (#19875130)
> As always Blockchains are a good solution only for a tiny group of problems.

Flowchart here: http://philippe.ameline.net/images/ShouldYouUseBlckchn.jpg

Re:Technical Details on the Recent Firefox Add-On Outage by dwaite on Friday May 10, 2019 @12:03AM (#19874948)
An OS owns and can enforce policies around the filesystem. For example, a future release of macOS might say that files cannot be both executable and writable, and the only way a file can be marked executable is by the security system validating a signature, or an administrative user override.

Firefox has to deal with malware injecting extensions outside the normal browser process, so I'm not surprised they would default to having periodic re-checks.

Re:Technical Details on the Recent Firefox Add-On Outage by michrassena on Friday May 10, 2019 @11:30AM (#19878685)
One small note on your mention of MacOS, and I don't know if this has been fixed. A couple of years ago, the certificates expired on some versions of the OS installer, like El Capitan and could no longer be verified. While there's the simple workaround of changing the clock, it did give me pause when trying to get an old Mac upgraded to the latest supported operating system.
Re:Technical Details on the Recent Firefox Add-On Outage by SyneRyder on Friday May 10, 2019 @03:52PM (#19881615)
On a similar note, I remember when Apple had a similar outage in the App Store [1], and a lot of my App Store bought software stopped working, claiming to be damaged. The workaround was to re-download the software again from the App Store, but because Apple doesn't let you download old versions, some of the apps had since updated to a newer macOS version than my machine could handle & I couldn't download them with the new certificates. Some of the smaller indie developers were gracious enough to send me non-signed non-App Store builds that I could keep using, but I did lose some software I'd bought. I haven't bought from the App Store ever since.

Considering Mozilla's certificate failure, Apple's cert failure, and the number of websites I encounter that have forgotten to renew their certificates, it seems like a broken system. Or a really effective form of DRM, I'm not sure which.

[1] https://discussions.apple.com/thread/7336980

Re:Technical Details on the Recent Firefox Add-On Outage by tinus_hn on Friday May 10, 2019 @04:51AM (#19876170)
How can you tel an app has actually been signed in the past? It could just as well been signed today with a fake date. This defeats the purpose of expiry.

You could just as well have certificates that never expire and just start signing with a new one if you feel like it.

Note the Mozilla ‘solution’ has the same problem: their root now authorized a new certificate with the old public key. If the original expectation was that the private key would be safe for a certain number of years (which is why you have expiry dates in the first place) the unsafe private key is now valid again. This defeats the purpose of expiry.

Re:Technical Details on the Recent Firefox Add-On Outage by michaelt on Friday May 10, 2019 @05:34AM (#19876319)

  It could just as well been signed today with a fake date.
The traditional option (used by things like Java and ActiveX since the early 2000s) is Trusted Timestamping [1], where a trusted third party provides a signature (such as via RFC 3161 Time Stamp Protocol). AFAIK every certificate authority that sells code-signing certificates also provides a free timestamping server. If you trust a CA to issue code-signing certificates, the theory is you'd trust them to timestamp too.

If you don't like having the trusted third party, you can also publish the hash in some write-once medium, such as on the blockchain, in certificate transparency logs, or in the small ads of a reputable newspaper.

[1] https://en.wikipedia.org/wiki/Trusted_timestamping

Re:Technical Details on the Recent Firefox Add-On Outage by tinus_hn on Friday May 10, 2019 @07:08AM (#19876697)
You don’t trust an authority, you trust a private key that matches a public key you know. The point is that as time marches on the key might leak or be cracked and then your trust is misplaced. So you say the key pair expires and you stop trusting it before that happens.

This trusted timestamping relies on the same signing things with public and private keys process so it just adds a step, it doesn’t solve the problem. The timestamping key needs to expire as well and then it can’t be trusted anymore.

Re:Technical Details on the Recent Firefox Add-On Outage by michaelt on Friday May 10, 2019 @09:49AM (#19877652)
Plenty of the CA root certificates in my browser's trust store have 30 year validity periods. The "keys should have short expiry periods" rule doesn't appear to apply to CAs.

Admittedly, CAs are held to higher standards than most certificate users in terms of keeping keys in hardware security modules and suchlike. So perhaps it's not 100% unjustified that they get longer validity periods.

Re:Technical Details on the Recent Firefox Add-On Outage by flohofwoe on Friday May 10, 2019 @06:08AM (#19876456)
The code-signing needs to happen through a trusted / certified timestamp server, it's not using the local timestamp. This process basically guarantees that the application used a valid certificate at the time it was signed, and that it wasn't tampered with since then.

With this, the time when code-signed application is executed doesn't matter at all, fake date or not.

Re:Technical Details on the Recent Firefox Add-On Outage by ngcc_hk on Thursday May 09, 2019 @08:37PM (#19873977)
Technically I think one must be clear. The cert (client and ca) all have dates and that you are right. But what your miss is that the revocation measure is the key here. It is not just sign it for the period. The CA and the root CA list has to maintain those cert and revoke them. You should not just trust the date and signature. The revocation list is key.

In other words you check everytime you use it basically. That is how stolen Microsoft cert stopped.

All old app are checked for validity. It is not as you implied.