The study likely assumed that people already familiar with working in these methodologies were equally available, equally happy to do so, and equally priced as regular developers, which is unlikely to be case. A certain minimum team size is probably required to make many of these processes viable; many real world systems are written by solo coders. Also, in reality it is relatively trivial to scavenge work like seL4, Linux kernel security toolkits, software HA clustering and/or proxy implementations, hardware watchdogs, ssss and/or distribute infrastructure across disparate locations to provide what probably amounts to magnitude higher redundancy and resilience against real world attacks or component failures without paying those costs. Indeed, the implementation of such is becoming near free many cases due to convergence on SoA/microservices.

To put this more plainly, "I can get another if I break it, so a clay cup trumps a grail." - Mirza Asadullah Khan Ghalib, classical Urdu and Persian poet from the Mughal Empire ... quote #1 in my fortune db @ https://github.com/globalcitizen/taoup

"The study likely assumed that people already familiar with working in these methodologies were equally available, equally happy to do so, and equally priced as regular developers, which is unlikely to be case."

For things like Cleanroom and PSP/TSP, they were first-time users of the process. Not long-time. Just decent developers shown a better way of getting results. In high assurance, it varied from first-time to experienced users. First time users usually gained the benefits of the method but with difficulty. There'd be at least one specialist aiding the team. Best results came from people who had done it a while. So, for high that's partially true and for stuff I push on commercial sector it's the opposite.

"A certain minimum team size is probably required to make many of these processes viable; many real world systems are written by solo coders. "

That's a legit critique. They can still benefit as I experienced myself and some others. It requires one to divide the work into periods where you wear various hats. It takes discipline to externalize one's own work and attack it objectively. Good news is that inspection, testing, static analysis, model-checking, design-by-contract... these all catch problems that are objective more than subjective. Still get plenty of results. There's been one or two threads on here where even solo people do QA on their own stuff with good results.

Still open-ended problem worth some experiments to see how much can be done at what effectiveness for a solo coder juggling work. So, good point.

"in reality it is relatively trivial to scavenge "

It's so trivial we had today yet another HN thread about distributed locks showing how people were copying one implementation and screwing it up. Then there was disagreement in the comments. There's threads like that on various critical things at least once a day or so on HN. Shows it's trivial to do and hard to do right for many things. Better to make re-usable, strong solutions to the various problems we keep encountering where possible. Can always mix high, medium, and average assurance components as a compromise where we have X goals but only Y resources. Pieces can improve over time.

"Indeed, the implementation of such is becoming near free many cases due to convergence on SoA/microservices."

I've yet to see one go 5 years under heavy load without downtime much less the 17 years VMS clusters did or 30+ years for some IBM mainframe installations. For security, things are more abysmal. I'll agree many solutions are popping up that can do amazing things. Robust, scalable, fairly secure, and near-zero-downtime systems don't seem to be among them in this nearly-free, mainstream market you're talking about. A rarity in the market in general.

So, there's still room for both medium and high assurance software. To be clear, I'm only pushing medium because even first-timers did a better job than most do today by eliminating the problem areas & doing great detection. Similar cost, too. Only hoping for high-assurance to be selectively applied to high-impact stuff that doesn't change often. A recent example was a formal verification of SCOOP: Eiffel's race-free, easy-to-analyze, concurrency model. It found issues in the model that were fixed. Three other analysis, ranging from specific to general, eliminated deadlocks, livelocks, and performance-hit respectively. That's the kind of situation where a significant investment of resources can keep providing benefits with almost no effort. Otherwise, medium assurance aka commercial + sane extras.

""I can get another if I break it, so a clay cup trumps a grail.""

Nice quote and good principle in much of current economy. Not so much my personal data or business I.P. when a byzantine failure takes out it and my hot standby. I think the comparison mainly applies to non-critical services and hardware. ;)

Interesting discussion, I'd be keen to try such a formalistic approach if you ever have a need for a remote critical eye.

Regarding the quote, I don't think you quite took the spirit of it: the idea is that, given a replaceable component with failure expected, we can manage reasonably to achieve the same outcomes for less overall cost - ie. do not build the one true system to solve all availability and security challenges (which would be very expensive (the old "pick two" adage)), but rather build a system which makes use of the properties of components intelligently to reduce overall hassle (as measured in thought, hassle, time, cost, third party dependencies, untested code or processes) to achieve the same objectives, just as reliably/securely.

I think the main reason there are less modern systems running for 15+ year timeframes is that the centralized models of yore only remain valid within traditional large organizations that are slow moving: finance, government infrastructure, military, etc. Even there, they are being dismantled, as COTS hardware + FOSS topple traditional mainframe vendors through cheap and highly scalable clustering. The paradigm has shifted: today we have components like the Linux kernel and sqlite which despite not generally running a single instance for >15 years (though they are definitely capable, and with Linux it has been done; sqlite is not old enough yet!) they do run in harsh environments with limited resources in hundreds of millions - nay - billions of instances, extremely reliably.

"Interesting discussion, I'd be keen to try such a formalistic approach if you ever have a need for a remote critical eye."

I'll try to keep it in mind. There's a few people doing stuff in public if you want to play with it. Plus lists to stuff to try. Here's a few.

One person did an overview of Cleanroom and examples in Python, etc here:

http://infohost.nmt.edu/~shipman/soft/clean/

I just pulled a list of techniques that were empirically proven to increase robustness into a Pastebin:

http://pastebin.com/xZ6m4T8Z

Note: One can pick and choose among these to find right work/benefit ratio. I'd like to see lots of experiments on mainstream software to get data on what's 80/20 rule. I'm betting on sane subsets of safe languages, code reviews, usage-driven testing, interface checks (even asserts), and fuzz testing maybe. The first four are widely available and take little work.

Case study of Altran's Correct-by-Construction approach showing great metrics:

https://www.commoncriteriaportal.org/iccc/7iccc/t1/t1201100....

Note: Altran does plenty of medium and high assurance work. Their method can do either. I'd rate Tokeener as Medium-High if just focusing on their part of it. Note that they got it done with 3 part-timers in less than a year with code-level proofs. Quite a leap from millions spent on teams of geniuses doing proofs in Gypsy, etc two years at a time back in the day. :)

"Regarding the quote, I don't think you quite took the spirit of it: the idea is that, given a replaceable component with failure expected, we can manage reasonably to achieve the same outcomes for less overall cost "

I'm not questioning the spirit so much as the implementation.

" do not build the one true system to solve all availability and security challenges (which would be very expensive (the old "pick two" adage)), but rather build a system which makes use of the properties of components intelligently to reduce overall hassle"

That's what I advocated. With medium assurance, it cost almost nothing extra so apply it to the components. With high assurance, if used at all, it can be used on just the most important and is by some niche players with good cost/benefit results. It's just that...

"to achieve the same objectives, just as reliably/securely."

...this has rarely proven to work. There's an old rule that you can't make high robustness systems by composing low-robustness components. They experience both individual and byzantine failures. That's why we see the companies building on them experiencing downtime that shouldn't happen on top of unavoidable downtime which I don't judge. So, certainly break the problem into smaller ones and compose simple (or cheap) solutions where possible. You just have a baseline of quality you need for the components. Commonly expressed as "weakest link" or "Garbage In, Garbage Out."

"I think the main reason there are less modern systems running for 15+ year timeframes is that the centralized models of yore only remain valid within traditional large organizations that are slow moving: finance, government infrastructure, military, etc."

Someone else speculated the same thing. There's some truth to it. Yet, there's overlap between the kinds of services running on them and in the modern, downtime-prone infrastructures. The stuff works on robust systems of old times but fails on new ones. What's the key difference? One was designed to be robust and one is a composition of pieces of known to unknown quality. Garbage in, garbage out.

"as COTS hardware + FOSS topple traditional mainframe vendors through cheap and highly scalable clustering."

They're definitely cheaper and faster. That's been the case since Beowulf clusters I built back in the day. Yet, their uptimes still haven't reached what OpenVMS hit in the 80's with bullet-proof clustering. Are you truly saying hardware and admin labor is so free right now that we can just throw dozens to hundreds of servers in 3-5 locations at every businesses' problems while ignoring the architectural, interface or code quality of any of it? I'll take QNX 2 here and 2 there over a bunch of Linux boxes any day so long as budget will allow the license. I'll have predictability, performance, self-healing, damage containment, and live updates all in a few megs of well-written code w/ POSIX apps.

See the difference between that and mainstream systems? Good architecture still matters. System/38 (AS/400), OpenVMS, NonStop, QNX, BeOS... all great architecture that squeezed way more out of any given piece of hardware, software, or admin effort. OpenVMS was also an example of medium assurance that was point of our discussion. Their Red/Blue iterations did 1 week development & test writing then 1 straight week of review and fixes with regression tests over weekends. Results were boxes so reliable admins sometimes forgot how to reboot it and such. Still not common on Windows and Linux boxes despite the great strides in reliability that nearly a billion dollars worth of labor bought those bad architectures. I like to poke at Linux folks that MINIX 3 eliminated most crashes in a few years with a small team while UNIX architecture took decades to mostly do the same thing. Architecture & assurance activities matter.

"though they are definitely capable, and with Linux it has been done"

I'd like to see evidence of that. The mainframes, AS/400's, VMS, NonStop's... these run all sorts of workloads without software repairs for years. Heavy workloads. I've investigated numerous claims of individual Linux boxes doing the same. It was almost always someone's barely-utilized, hobbyist web server, DHCP server, etc. I want to see the equivalent of a mainframe or VMS server running everything in a business 24/7 without a crash in years. Optionally that system getting lost due to never needing maintenance with people scouring the building looking for it. That was a recurring problem with pizza box servers and embedded cards running aforementioned OS's. ;)

"they do run in harsh environments with limited resources in hundreds of millions - nay - billions of instances, extremely reliably."

Again, a source for this. They certainly run regular workloads they're optimized for quite reliably. The cloud vendors optimizing on specific HW configurations with all kinds of disaster tolerance appear to run them reliably by hiding the failures. Yet, surveys of industry on their servers' downtime contradicts the overall claim with all kinds of downtime still in mainstream stuff. Plus, they show recent Windows servers ahead of Linux in reliability and AIX usually stayed on top. They don't include the stuff I mentioned due to maybe market share or perhaps the contrast would make the others look bad. ;)

Nonetheless, adopting medium assurance methods to build our components or using components built that way is a solution. It was proven... is used... in many commercial systems that have higher quality at acceptable price and performance. Outside one-offs, the only reason it's not happening is lack of knowledge or will to do it. Mainstream devs were told about the stuff for years so they just don't give a shit. Surprisingly, Microsoft is ahead in efforts as they implemented SDL, formal methods on drivers, integrity controls, app protections, safe language use, and so on. I would love to see that level of commitment on BSD and Linux side given I use both. Least they sometimes accept fixes from academics who apply various tools in their spare time or with grant funding. ;)

The pastebin looked fine but an issue may be that some of the data/samples in supporting research are so old as to be difficult to apply to modern infrastructure, and the HR question still stands.

I will take a look at http://infohost.nmt.edu/~shipman/soft/clean/ tomorrow (evening here).

I agree with your assertions around reasonable use of prior work. I personally came to similar methods designing security-oriented systems (eg. at Kraken), eg. by strictly defining a protocol in a formalistic manner with a language suited for that purpose, then generating interfaces for client systems ... and even application-level enforcement proxies that can be inserted either during testing or (preferably) even automatically during live deployment. This amounts to automated application-level middleware, for SOA-type systems, direct from protocol spec.

Enforcement proxies ensure that broken implementation A and broken implementation B may only talk correct subset C of any given protocol, block and alarm on noncompliant communications, make otherwise restricted communication between nodes extremely difficult for attackers to use to horizontally expand any foothold established through security breaches. Assertions may be enforced in real time. Logging may occur there also. Very useful stuff. Not actually that hard to do, in practice. By contrast, the current norm of arbitrary HTTP POST with barely enforced JSON is a leaky bucket of slop.

Altran link 404s for me.

The internet's basis in decentralization roughly equates to FOSS clustering equates to torrents equates to the wisdom and utility of failable components, I do not think you can claim in good faith that embracing failure has "rarely proven to work".

Therefore, I'd like to know the source of your "old rule" that you can't make high robustness systems by composing low-robustness components, because in the face of just the above evidence it looks like bullshit. Get enough redundancy in a system and it will operate with high resilience, as long as the orchestration algorithms are well tested. Redundancy is virtually free today in hardware, software, network connectivity, legal jurisdiction and a host of other ways. Not to take advantage of this reality is to have one's head firmly in the sand. To take advantage brings benefits beyond merely security and availability, such as system mobility, peer service provider independence, legal jurisdiction independence, cheaper sourcing, easier scaling, etc.

Some of the rest of your post seems to me to be comparing apples and oranges, bygone eras to the present, monolithic transaction-oriented mainframes to general purpose complex COTS disposables, so I don't feel it warranted to discuss further.

With regards to Linux uptimes, you never specified workload, and that's hard to assure in hindsight, so you've created an impossible requirement to disprove your assertion. You do recognize however that many standard load Linux systems have run for 5+ or 10+ years with no problem, as I too know well as I've had some. Further, with changes in hardware (more work in RAM than ever, less unpredictable disk failure, higher quality cooling, more recognition of component lifetimes, ECC RAM, etc.) I expect that net COTS system longevity is increasing rather than decreasing, and Linux will not crash before the hardware in all but very unlucky cases... particularly with easily scripted, reasonable load testing.

My quote about sqlite and Linux running in harsh environments with limited resources in hundreds of millions - nay - billions of instances, extremely reliably, was primarily referring to mobile devices. Power challenges, many power cycles, physical bashing, temperature variations, heavily intermittent connectivity, dynamic clock speed migrations, moisture, etc. Furthermore, they are far more complex in terms of software stack than 1980s/early 1990s transaction processing oriented mainframes.

" may be that some of the data/samples in supporting research are so old as to be difficult to apply to modern infrastructure"

They still apply based on similar stuff published since then. Several are best practices for robust systems even today. The HR question should've been addressed by fact that medium assurance techniques take either little or no training: mostly just effort and accountability.

"This amounts to automated application-level middleware, for SOA-type systems, direct from protocol spec."

Great work. An example of the higher end of what I'm talking about. I did something similar and kept checks on in production. This lets me laugh about these obvious vulnerabilities like protocol downgrades while others scurry around trying to patch them. Common problem that tools can help us do better so we use tools to do it better. I'd like to see more think that way.

I cleared history and tried Altran link again. It works on my end. Try this one again:

https://www.commoncriteriaportal.org/iccc/7iccc/t1/t1201100....

Note: I'd really rather give you one that shows the use of the method like on the CA they built. Yet, the Altran transistion and effect of time means links are disappearing off the net slowly. Happened to old language studies, too, that weren't put into IEEE or ACM.

"The internet's basis in decentralization roughly equates to FOSS clustering "

Not quite. When a site goes down, it goes down. I have to manually look for mirrors, archive.org, etc. One can use load balancers, CDN's, etc to cluster such things. That's not default, though.

" equates to torrents"

The default at the time was FTP or HTTP. This led to problems. Following my recommend, a person sat down to think of how they'd meet their goals differently. A combo of good architecture and coding led to BitTorrent.

"I do not think you can claim in good faith that embracing failure has "rarely proven to work"."

My overall claim is that you benefit by taking extra time to apply lessons learned to architecture, design, and coding to increase effectiveness (esp quality). The two cases you cited support my point that this creates benefits rather than your original case of cheap labor throwing together whatever they can download. Most people who are stitching together components with good long-term results are using components or cookbooks made with my approach. So one may even precede the other.

Far as embracing failure, high assurance has been on that for a while. Hell, the concept even presupposes failures across the lifecycle. Most such systems are designed as interacting FSM's that incorporate failure states with a fail fast and fail safe approach. Later on, it was taken further with "recovery-oriented computing" funded by DARPA, etc. Google that phrase with words "survey" and/or "security" for some interesting architectures.

"I'd like to know the source of your "old rule" that you can't make high robustness systems by composing low-robustness component"

It came Bell of Bell-Lapadula model. Probably another person, too, though can't recall name. The DOD funded high assurance safety & security research leading up to Orange Book A1 systems over a period of three decades. Some systems were built to right principles from the start. Some tried to retrofit things like UNIX for both availability and security. All the security plus some availability retrofits failed with pentesters finding more and more obscure failures or vulnerabilities. These often resulted from interactions of overly complex, stitched-together components. That's because combining two complex programs essentially makes a third, complex program. Being unable to prove reliability or security of input components makes it difficult to say anything about output.

Note that this is true for security more than availability. HA solutions show that, depending on failure type, one can get at least three 9's out of crap systems. The HA had to be well-designed and coded. Fault-tolerance took solutions like NonStop or Stratus straight designed for them. So, both took my model to varying degrees either at middleware or whole-system level for availability. Many of these still failed at upper layers of stack, occasionally lower ones, due to Byzantine failures due to how data moves throughout the cluster & corrupts stuff. Led to Byzantine-tolerant schemes, recovery-oriented architectures, and MILS architecture for secure composition of stuff like Linux. These efforts had mixed success with the problem hard enough that DARPA & NSF are still funding smart people trying to find & prove anything about a solution up-front.

So, I think my claim is well-grounded in evidence. Availability often applies on common cases that really smart people... carefully designing and coding as I advocate... optimized the hell out of. Same at various layers of the stack. These things work well enough that average users will get results unless they stress them unexpectedly. Others are solving root problems where they can with better systems and software. Think RDBMS clusters vs Hadoop stuff vs Google's F1 RDMBS. The best method is composing a combo of what exists... in common, battle-hardened configuration/usage... with higher-quality stuff while incrementally replacing what you can as better stuff appears. This was high-assurance strategy called "incremental MLS" designed to address high costs and slow time-to-market of full, custom MLS. Worked wonders applied to web servers, databases, etc.

" You do recognize however that many standard load Linux systems have run for 5+ or 10+ years with no problem, as I too know well as I've had some. "

Most I've seen didn't. Yours did. Just collecting data to support or refute their claim. It's certainly possible. The question is how probable. It's close to 100% for some solutions in real-world loads. Those are pricey for sure but they work by design that can be emulated. I'd like to know where Linux's design & code puts it.

"comparing apples and oranges, bygone eras to the present, monolithic transaction-oriented mainframes to general purpose complex COTS disposables"

The 2016 mainframes, NonStop clusters, IBM i's, etc run similar workloads with high availability and throughput on mainframes due to Channel I/O. The modern COTS "disposables" in cloud sector are even setup to emulate mainframes with many CPU's, virtualization, metered CPU/memory use, some I/O offloading, management software, etc. It's getting more apples to apples all the time. Their reliability just still isn't there with whole datacenters going offline here and there. The management tech, on other hand, has caught up in the lights out datacenters maybe even exceeded prior work.

You want to talk disposables, though. So, let's compare good architecture on a budget vs common architecture. In embedded sector, there's uptake of Java subsets to run safety-critical applications. Common deployments are microcontrollers or PowerPC boards with weak CPU's running a watered down JVM. The JOP people and Sandia's SSP team each just built a JVM on silicon w/out any abstraction issues that would hurt them. Everything down to OS code is Java bytecode. Results were low cost, low watts, acceptable performance, and high safety. The Azul Systems Vega processors did something similar for premium, enterprise market with hardware-assisted pauseless GC and tons of cores doing native Java. A fraction of that could be implemented inexpensively with standard cell model and sold as plug-in cards to disposable market.

https://www.azul.com/products/vega/

There's doing what's typical but keeps causing problems vs doing something different that addresses needs & root problems from ground up. Not always an option but is more than people would admit.

"I expect that net COTS system longevity is increasing rather than decreasing"

That's debatable. You cited good evidence to back it up. The counter evidence is that the ridiculous levels of complexity in modern hardware and problems on deep-submicron nodes mean things fail in little ways more often. Overall, your claim holds as hardware is very reliable given complexity level and they've forced the failures into a few, predictable spots. We occasionally get surprises like RAM fault paper. Yet, clustering covers basic stuff pretty well.

"My quote about sqlite and Linux running in harsh environments with limited resources in hundreds of millions - nay - billions of instances, extremely reliably, was primarily referring to mobile devices."

You should've said that. Yes, embedded stuff is easier to do that with. Lots of embedded devices run for years. Especially with clock gating etc where most of the chip stays off most of the time only coming online to do its work. Quite effective in mitigating transient errors. Let's see. My Galaxy freezes up on cameria here and there. Just failed to recognize SIM card yesterday requiring battery reset. Otherwise, very reliable as you say given complexity of stack.

Yet again, as it's almost a trend in this discussion, were SQlite and Linux done in the cheap/free people stich together whatever exists method you espoused or clean-slate solution that carefully designed or reviewed changes as I push? I'm cheating as I know Linux kernel process is pickier & lower defect than average. I also know SQLite has a strong review process designed to keep complexity to a minimum and quality up. Two more examples leveraging my recommendations. :P