Safety critical systems are not a good fit for a blockchain-based resolution to the Byzantine General problem. Safety critical systems need extremely low latency to resolve the conflict fast. So blockchain is not going to be an appropriate choice for all critical applications when there are multiple low-latency solutions for BFT at a very low ms latency and IIRC, microseconds for avionics systems.

Not sure what you mean with "safety critical". I made no such assumption. Also since the topic is about a (write-only) log it probably doesn't need such low latency. What it more likely needed is final states so once a entry is made and accepted it must be final and ofc correct. DLTs can do this distributed and self-fixing i.e. a node that tries to add fault data is overruled and can never get a confirmation for a final state that later would not be valid.

Getting the whole decentral system to "agree" will never be fast unless we have quantum tech. There is simply no way servers around the glob could communicate in microseconds even if all communication would be the speed of light and processing would be instant. It would still take time. In reality such system need seconds which is often totally fine. As long as everyone only relies on the data that has been declared final.

You said If a system is critical

I thought you were making a more general statement about all critical systems, that's all. And since many critical systems have a safety factor in play, I wanted to distinguish them as not always being a good target for a Blockchain solution to the problems of consensus.

Blockchain is a very interesting solution to the problem of obtaining consensus in the face of imperfect inputs, there are other options so, like anything else, you choose the right tool for the job. My own view is that-- given other established protocols, blockchain is going to be overkill for dealing with some types of fault tolerance. It is a very good fit for applications where you want to minimize relying on the trust of humans. (And other areas too, but right now I'm just speaking of the narrow context of consensus amid inconsistent inputs)

Critical that the system operates/keeps operating/does not reach an invalid state. It could be for safety but in general its more to avoid financial damage. Downtime of any kind usually result in huge financial loses and people working extra shifts. This was my main point.

>...blockchain is going to be overkill for dealing with some types of fault tolerance

But it this case likely it isn't. The current systen already works with a chain of blocks its just lacks the distributed checking and all that stuff. But "blckchains" aren't some secret sauce in this case it just an way to implement a distributed write-only database with filed proven tech. It can be as lightweight as it any other solution. The consensus part is completely irrelevant anyway because all nodes are operated by one entity. But due to the use case (money/value) of modern DLT ("blockchains") they are incredible reliable by design. The oldest DLTs that uses FBA (instead of PoW/PoS) are running since 9+ years without any error or downtime. Recreating a similar reliable system would be month and month of work followed by month of testing.

Yep, pretty much agreed. Whatever anyone may think if crypto currencies blockchains are essentially a different technology where coins are just 1 application. I'm kind of "meh" on crypto currencies (not anti, just think they need a while more to mature) but trustless consensus is a significant innovation in its own right.

I haven't thought about whether an actual blockchain is really the best solution, but the redundancy argument is legitimate. We've been doing it for decades in other systems where an unnoticed bit flip results in complete mission failure, such as an Apollo mission crash.

I'm not really sure what Yeti 2022 is exactly, so take this with heaps of salt, but it seems like this is a "mission failure" event -- it can no longer continue, except as read only. Crypto systems, even more than physical systems like rockets, suffer from such failures after just "one false step". Is the cost of this complete failure so low that it doesn't merit ECC? Extremely doubtful. Is it so low that it doesn't merit redundancy? More open for debate, but plausibly not.

I know rockets experience more cosmic rays and their failure can result in loss of life and (less importantly) losing a lot more money, and everything is a tradeoff -- so I'm not saying the case for redundancy is water tight. But it's legitimate to point out there is an inherent and it seems under-acknowledged fragility in non-redundant crypto systems.

>I haven't thought about whether an actual blockchain is really the best solution...

Most likely not. But the tech behind FBA (Federated Byzantine Agreement) distributed ledgers would make an extremely reliable system that can handle malfunction of hardware and large outages of nodes. And since this is a write-only log and only some entities can write to it, it could be implemented with permission so that the system doesn't have to deal with attacks that public blockchain would face.

Technically everyone can write to it. However you can only write certain specific things.

In the case of Yeti 2022 you were only able to log (pre-)certificates signed by particular CAs trusted in the Web PKI, which were due to expire in the year 2022.

In practice the vast majority of such logging is done by issuing CAs, as part of their normal operations. But it is possible (and is done purposefully, at least sometimes) to obtain certificates which have not been logged. These certificates, of course, won't work in Chrome or Safari because there's no proof they were logged. But you can log them yourself, and get SCTs and show those Just In Time™.

This is only an interesting technical option if you have both the need to do it for some reason and the capability to send SCTs that aren't baked inside your certificates. The vast majority of punters just have the CA do all this for them and the certificate they get has SCTs baked right inside it so there's no technical changes for them at all, they needn't know CT exists.

Because the CA's critical business processes depend on writing to logs, they need a formal service level agreement assuring them that some logs they use will accept their writes in a timely fashion and meet whatever criteria, but you as an individual don't need this, you don't care if the log you wanted to use says it's unavailable for 4 hours due to maintenance.

Thats pretty much the default state of any blockchain like systems. You need a private key to write to it. Its just that in most public blockchains can have an infinite amount of new private keys can be generated an and some kind of token is attached to it. For a log none of that would be needed. A central operator could hand out keys to anyone who should be able to write to it and for all other its read-only. And ofc the key alone would still not allow someone to write invalid data.

> I'm not really sure what Yeti 2022 is exactly

Sometimes there are problems with certificates in the Web PKI (approximately the certificates your web browser trusts to determine that this is really news.ycombinator.com, for example). It's a lot easier to discover such problems early, and detect if they've really stopped happening after someone says "We fixed it" if you have a complete list of every certificate.

The issuing CAs could just have like a huge tarball you download, and promise to keep it up-to-date. But you just know that the same errors of omission that can cause the problem you were looking for, can cause that tarball to be lacking the certificates you'd need to see.

So, some people at Google conceived of a way to build an append-only log system, which issues signed receipts for the items logged. They built test logs by having the Google crawler, which already gets sent certificates by any HTTPS site it visits as part of the TLS protocol, log every new certificate it saw.

Having convinced themselves that this idea is at least basically viable, Google imposed a requirement that in order to be trusted in their Chrome browser, all certificates must be logged from a certain date. There are some fun chicken-and-egg problems (which have also been solved, which is why you didn't need to really do anything even if you maintain an HTTPS web server) but in practice today this means if it works in Chrome it was logged. This is not a policy requirement, not logging certificates doesn't mean your CA gets distrusted - it just means those certificates won't work in Chrome until they're logged and the site presents the receipts to Chrome.

The append-only logs are operated by about half a dozen outfits, some you've heard of (e.g. Let's Encrypt, Google itself) and some maybe not (Sectigo, Trust Asia). Google decided the rule for Chrome is, it must see at least one log receipt (these are called SCTs) from Google, and one from any "qualified log" that's not Google.

After a few years operating these logs, Google were doing fine, but some other outfits realised hey, these logs just grow, and grow, and grow without end, they're append-only, that's the whole point, but it means we can't trim 5 year old certificates that nobody cares about. So, they began "sharding" the logs. Instead of creating Kitten log, with a bunch of servers and a single URL, make Kitten 2018, and Kitten 2019, and Kitten 2020 and so on. When people want to log a certificate, if it expires in 2018, that goes in Kitten 2018, and so on. This way, by the end of 2018 you can switch Kitten 2018 to read-only, since there can't be new certificates which have already expired, that's nonsense. And eventually you can just switch it off. Researchers would be annoyed if you did it in January 2019, but by 2021 who cares?

So, Yeti 2022 is the shard of DigiCert's Yeti log which only holds certificates that expire in 2022. DigiCert sells lots of "One year" certificates, so those would be candidates for Yeti 2022. DigiCert also operate Yeti 2021 and 2023 for example. They also have a "Nessie" family with Nessie 2022 still working normally.

Third parties run "verifiers" which talk to a log and want to see that is in fact a consistent append-only log. They ask it for a type of cryptographic hash of all previous state, which will inherit from an older hash of the same sort, and so on back to when the log was empty. They also ask to see all the certificates which were logged, if the log operates correctly, they can calculate the forward state and determine that the log is indeed a record of a list of certificates, in order, and the hashes match. They remember what the log said, and if it were to subsequently contradict itself, that's a fatal error. For example if it suddenly changed its mind about which certificate was logged in January, that's a fatal error, or if there was a "fork" in the list of hashes, that's a fatal error too. This ensures the append-only nature of the log.

Yeti 2022 failed those verification tests, beginning at the end of June, because in fact it had somehow logged one certificate for *.molabtausnipnimo.ml but it has mistakenly calculated a SHA256 hash which was one bit different and then all subsequent work assumed the (bad) hash was correct. There's no way to rewind and fix that.

In principle if you knew a way to make a bogus certificate which matched that bad hash you could overwrite the real certificate with that one. But we haven't the faintest idea how to begin going about that so it's not an option.

So yes, this was mission failure for Yeti 2022. This log shard will be set read-only and eventually decommissioned. New builds of Chrome (and presumably Safari) will say Yeti 2022 can't be trusted past this failure. But the overall Certificate Transparency system is fine, it was designed to be resilient against failure of just one log.

Conceivably you could also fix this by having all verifiets special case this one certificate in their verification software to substitute the correct hash?

Obviously that's a huge pain but in theory it would work?

You really want to make everyone special case this because 1 CT log server had a hardware failure?

This is not the first time a log server had to be removed due to a failure, nor will it be the last. The whole protocol is designed to be resilient to this.

What would be the point of doing something besides following the normal procedures around log failures?

I was just asking to check if I understood the problem correctly.

Yes, in principle you could special case all the verifiers, although that's an open set, the logs are public.

Thank you, that's an excellent description. The CT system as a whole does appear to have ample redundancy, with automated tools informing manual intervention that resolves this individual failure.