News for hackers. Stuff that matters.
Re:This shouldn't have happened: A vulnerability postmortem by lucb1e on Wednesday December 01, 2021 @02:38PM (#29408162)
A title that actually describes the post, mostly paraphrasing the first paragraph:

Reasons why this buffer overflow wasn't caught earlier despite doing all the right things

And then to give those reasons:

- "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could never be discovered."

- "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes"

- "combined [fuzzer] coverage metrics [...]. This data proved misleading, as the vulnerable code is fuzzed extensively but by fuzzers that could not possibly generate a relevant input."

The conclusion is, of course, to fix those problems if your code base also has them, but also "even extremely well-maintained C/C++ can have fatal, trivial mistakes".

Re:This shouldn't have happened: A vulnerability postmortem by titzer on Wednesday December 01, 2021 @10:54PM (#29412387)
The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language."

No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.

Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @12:52AM (#29412979)
No issue with the first sentence of your message at all, but...

> No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.

We won't wonder at all. We will understand that those people are the ONLY ones that stepped up to the task over 50 years to write this kind of software, organize standards bodies for their languages and platforms, get their software packaged as part of mainstream operating systems and out into the world, deal with patches from users, and help millions of other people make a living, enable the internet to happen, etc.

We will wonder why with all the millions of lines of C/C++ reference code available to be perused and then rewritten in Rust, Pascal, C#, Zig or Nim, and the vociferousness of their advocates, why that didn't happen in a reasonable timeframe.

We will wonder why all the Lisp and Haskell programmers who sneer down their nose at working C/C++ programmers in forums like this on a daily basis, didn't get off their asses with their One True Language (TM) and come to the worlds rescue.

The answer will be: these people aren't doers - they are talkers. It's one thing to get a toy proof of concept OS in your favourite language of choice that supports like 5 drivers. It's another thing to contribute to and build an ecosystem depended on by millions of developers daily. C/C++ people may not always be the sharpest tools in the shed, and they may be a dime a dozen. But they know how to organize themselves in loose groups of more than just a few developers, and work with other people.

We may have issues with the quality of what they ship on a frequent basis, but at least they ship instead of posting endless comments on internet forums about how it's all the others peoples fault.

Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @02:50AM (#29413575)
> We will wonder why with all the millions of lines of C/C++ reference code available to be perused and then rewritten in Rust, Pascal, C#, Zig or Nim, and the vociferousness of their advocates, why that didn't happen in a reasonable timeframe.

Easy, every single time .NET team does some advances into that direction, it gets sabotaged by WinDev and their C++ love.

XNA vs DirectXTK, .NET vs COM/WinRT,...

Windows could have turned into something like Android, with managed userspace and a very constrained native layer for restricted use cases, naturally WinDev cannot let that ever happen.

Re:This shouldn't have happened: A vulnerability postmortem by flohofwoe on Thursday December 02, 2021 @05:05AM (#29414359)
That was simply because the alternatives to Win32 had various serious regressions, be it usability, features, bloat, or other things that are important in one way or other to the people who stick with Win32.

More secure languages and APIs can only win if they are both easier to use and offer the same features.

Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @05:12AM (#29414404)
As proven by mobile OSes, it is not technical features that win the game, rather railing the developers into the future, regardless of their opinion.

The problem at Microsoft is that what Office and WinDev want, drives the whole business no matter what, those "serious regressions, be it usability, features, bloat, or other things" get fixed, if one cares enough to make it happen.

Speaking which, Office now is an heavy user of JavaScript, less perfomant than .NET, because it needs to be in the cloud.

Whereas Windows, while king of the desktop, is undeniable that it lost the backend to POSIX clones even if we consider MS shops with IIS, where languages like Java and Go dominate.

Also apparently PWAs are cool now as fight against ChromeOS, which from performance point of view are much worse than those former Win32 alternatives.

Re:This shouldn't have happened: A vulnerability postmortem by blub on Thursday December 02, 2021 @03:47AM (#29413882)
Before I ditched Windows a couple of years ago, I was able to experience first hand how bloated and slow was the software that Microsoft rewrote in C#, so I kind of understand why such rewrites were being sabotaged.

If by some minor miracle a C# or Java GUI app is not slow, then it will use a ton of memory. A whole OS of such apps would be a nightmare.

Re:This shouldn't have happened: A vulnerability postmortem by ptx on Thursday December 02, 2021 @04:25AM (#29414113)
The new modern/UWP apps in Windows 10, like the calculator and start menu, are written in C++, aren't they? They manage to be horrendously slow and bloated without C#, so maybe C# isn't the problem.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @05:08AM (#29414376)
They are, UWP is basically another take on what was being discussed before .NET as COM evolution, and Longhorn's failure has given them the wind to pursue it as Windows foundation.

So since Vista all major Windows APIs are COM based, not always surfaced to .NET, and we are expected to just go through books like ".NET and COM: The Complete Interoperability Guide" and do the needful ourselves.

WinRT as introduced in Windows 8 was then the full reboot, with .NET metadata taking over TLB files (COM type libraries) and introducing a new base interface IInspectable.

So the long term vision pursued by Synofsky, was that .NET Native and C++/CX would take over with COM fully replacing .NET.

Naturally it all failed down during the borked execution, and now you still have devs pushing for C++/WinRT, the C++/CX replacement, with ATL like tooling as "modern". Maybe it is modern given the tools that they are used at WinDev, I guess.

Ars even has a nice article how this reboot took place,

https://arstechnica.com/features/2012/10/windows-8-and-winrt...

Re:This shouldn't have happened: A vulnerability postmortem by flohofwoe on Thursday December 02, 2021 @05:10AM (#29414390)
It's quite simple really. Bad programmers (or good programmers in bad environments) are able to write slow code in any language, and the current Windows Desktop team seems to be an example of that.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @05:18AM (#29414431)
When people write worse C++ code than naive C# performs, it is really bad.
Re:This shouldn't have happened: A vulnerability postmortem by davedx on Thursday December 02, 2021 @04:25AM (#29414105)
You literally mention Java GUI apps in response to a post that calls out Android, a mobile OS and application ecosystem implemented in Java. The languages are not the issue.
Re:This shouldn't have happened: A vulnerability postmortem by blub on Thursday December 02, 2021 @10:21AM (#29417182)
It took Google more than a decade of trying and a VM rewrite to get close to the perceived performance of iOS.

And anything performance intensive was done in the NDK anyway.

All of this because of Java.

Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @11:11AM (#29417999)
Indeed, the big difference is that Google was willing to put money on the table to make it work.

Those improvements are exactly what WinDev sabotaged in regards to Windows.

Also in case you have forgotten, Objective-C and Swift are also managed languages, check chapter 5 of Garbage Collection Handbook, or any other CS reference in automatic memory management algorithms.

Re:This shouldn't have happened: A vulnerability postmortem by vips7L on Friday December 03, 2021 @04:08PM (#29435497)
Android is not Java.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @04:58AM (#29414324)
As proven by Midori (used to power Asian bing while in development), Android, ChromeOS, iOS/iPad OS, among others, it is possible when everyone works together for a common goal instead of sabotaging others work.
Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @08:19AM (#29415726)
I don't think this paints an accurate picture. This sort of presupposes that C/C++ were a first iteration, but that's not true.

People chose C and C++ for bad reasons, even with historic context. Languages used to be more correct. Algol2 used to validate array subscribting and not have null etc. It was C programmers who pushed languages to be worse because it made it easier to write code.

They very much created this problem and we regressed because of their choices. Things could have been a lot better.

Re:This shouldn't have happened: A vulnerability postmortem by shadowgovt on Thursday December 02, 2021 @09:34AM (#29416589)
I think it probably makes sense that the languages it's easiest to write code in would dominate in an ecosystem where incentives are encouraging software to eat the world.

The asbestos comparison is pretty apt. Possibly also apt would be steel plants. Yes, steel manufacture is extremely hard on the local environment. Doesn't matter. The world's in the middle of an industrial revolution, and we need steel now. Now now now. We don't have time to wait for the technology to catch up with environmentally-minimized-impact manufacturing. Just condemn a couple cities to twilight noon-times and get on with it.

I don't know if things could have been a lot better and we could be having this conversation on a decentralized network of anonymous machines like we are now. We had to conscript a lot of developers to get here, and make very complex software run on a lot of low-power devices. Erlang, as an example of an alternative, wasn't even open-source unil 1998. LISP machines paused for sixty seconds to do a GC cycle in an era contemporary with machines running executables written in C just... Not requiring that.

Given the factors involved (private ownership of language resources, complex runtime putting minimum requirements on hardware, etc.), C and C++ might be the only two paths of least resistance for a critical era of the software industry.

Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @09:38AM (#29416651)
> Given the factors involved (private ownership of language resources, complex runtime putting minimum requirements on hardware, etc.), C and C++ might be the only two paths of least resistance for a critical era of the software industry.

Maybe! We'll never know. I just don't think that rewriting history to be "C people were doers" is giving any meaningful context to language development, or how much was getting done in languages that were significantly safer.

Re:This shouldn't have happened: A vulnerability postmortem by shadowgovt on Thursday December 02, 2021 @10:10AM (#29417070)
I agree. The thing of interest here, I think, is the observation that language popularity is path-specific. JavaScript is just the worst, but everyone who does browser work has to know it because it happened to be the language clapped together to demo some nifty ideas for extending HTML and then those ideas stuck. Alternatives to JavaScript existed but were either privately-owned and therefore not trusted (vbscript) or too complicated to get off the ground (at some point, Mozilla had a fascinating proposal for HTML pages selecting features of the rendering engine and possibly even declaring that the whole page should be interpreted via user-definable code modules; had this won the day, you could write you web page in LaTeX for all anybody cared, but no-one was going to invest the time to write a dozen rendering agents when the HTML one was already expensive to develop and keep stable).

"C people were doers" is probably reductive, but C and C++ had an alchemy of factors that made something like, say, LISP or Erlang not be where they are.

If I had to hazard a guess, I'd say the most dominant factors are the language runtime's portability, the language's features allowing for relatively low-level access to the backing hardware (i.e. the thinness of the abstraction between C memory access and microprocessor read-write instructions means things like memory maps or "magic addresses" that trigger hardware effects on mutation could just be handed to user code), and the compatibility of the runtime with what came to be (for a ton of reasons) a dominant model of computation in the commercial and consumer space: the x86 architecture.

Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @10:16AM (#29417131)
I would guess that a huge part of C's popularity is UNIX. I think a number of other languages could have easily competed in the other areas ie: thin abstraction.
Re:This shouldn't have happened: A vulnerability postmortem by rstuart4133 on Friday December 03, 2021 @07:15AM (#29429129)
> People chose C and C++ for bad reasons, even with historic context.

Quite a bit of the original Unix's were written in assembler. There were good reasons for that - memory and CPU cycles were very scarce back then. I don't know if you have written assembler, but it takes about 5..10 times more lines of code than any high level language, and it isn't the nicest thing to read and isn't exactly portable between architectures.

C is an assembler without those problems, while retaining the speed. Admittedly t achieves that where Alogo60 didn't that by dropping minor things like bounds checking. But to give you a feel for the tradeoff, I could tell you what instructions the compiler would emit for most lines of C code. Best of luck doing that with an Alogo60 thunk.

C hit a sweet spot in other words and Algo60 didn't. C++ was originally called "C with classes". I was there at the time. The magic was Stroustrup's vtables, which gave C programmers OOP (which was all the rage at the time after Simula made it popular, and to be fair was a huge improvement on the abstractions C provided). It was that almost the same speed as C and to achieve it came with all the same disadvantages. But to Stroustrup's credit he did what he could - added stricter type checking (like function prototypes). It was an reasonable improvement. Many of Stroustrup's ideas were back ported to C, and C is a much better language for it.

Then Stroustrup added templates. In hindsight perhaps that's where the rot set in, but templates gave us zero cost abstractions that were at least as type safe as the original. It was an impressive achievement. But then there were hints of the darkness that was to fall upon us, with library after library using templates in novel very useful ways. Stdio was made typesafe. In so many ways.

C++ kept organically growing like that, into the nightmare it is today. I gather no C++ shop uses all of C++ now - all use some subset they can cope with. I lost interest ages ago, as did a lot of other people. They rebelled with against the complexity and lack of safety with things like Java. Microsoft rebelled against Java with C#. Those did solve the memory, type safety footguns and complexity of C++, but at the cost of the one thing about C that made it so attractive - low run time overhead and predictability.

In the mean time the ivory towers played with things like ML and Haskell, which with the benefit of hindsight was truly awesome work but unfortunately for those of us who work close to the metal are less practical than Java, C# and Javascript.

And then out of the swamp rose Rust, a language that has the speed of C and the type safety of ML and Haskell. Well sort of.

I still remember when I first read the Rust spec and thought they were kidding themselves - it promised so much and sounded so improbable it looked like a crypto ponzi scheme. Type inference,memory and thread saftey and no GC, what was this heresy? Then I dabbled, wrote a few Rust programs, fought the borrow checker until we make an accommodation, read the standard library and saw all those unsafes - and realised it was real. It was not perfect, but all those compromises to make it work is what reality looks like.

Where were we. Oh yes:

> Things could have been a lot better.

Things _are_ a lot better my friend. It just took us longer than expected to get there.

That wasn't because people made bad choices. They made a whole pile of small choices that solved their particular problem. It pains me to say this now, but after dealing with C++ for years anything would look good, and Java 1.0 did look very good to me. Admittedly it only looked good for a while, but I would never say the people who picked Java up at the time made a bad choice, just like I would never say the people who chose C over assembler made a bad choice.

Re:This shouldn't have happened: A vulnerability postmortem by FeepingCreature on Thursday December 02, 2021 @02:34AM (#29413493)
I don't think this is fair. First-mover advantage is absolutely a thing at the ecosystem level.
Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:31AM (#29415289)
There were plenty of first movers before UNIX and C, and later C++. They didn't stick. Now, step up to the plate and fix it.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @08:35AM (#29415885)
It is hard to stick against free beer OS, available with source code.

Had UNIX been a comercial endevour and it would have failed, unfortunely it wasn't the case, and now C plagues the industry.

Re:This shouldn't have happened: A vulnerability postmortem by Scramblejams on Thursday December 02, 2021 @04:12AM (#29414031)
> We will understand that those people are the ONLY ones that stepped up to the task over 50 years to write this kind of software

Not to take away from what they've shipped, but it bears repeating that they are also, generally, profligate writers of severe security holes. Some of the most expert programmers in the world have created mighty CVEs.

There's a school of thought that it's a poor craftsman who blames his tools, and your post walks close to that line, but when the tool is full of footguns even the experts can't avoid that cause tremendous damage to bystanders, then perhaps these people who know how to organize themselves in loose groups of more than just a few developers need to prioritize helping us all move to better tools.

Other segments of the programming population embrace a wild west culture of change, for good or bad. If the JS world can insist on a new frontend stack every few years, why can't managers and lead developers put their foot down and steadily back C/C++ into the corner where it belongs, where literally no other tool will do, and where it's doing the absolute minimum possible?

Since they're so good at shipping, ship us better tools! It may not seem a fair criticism -- it doesn't sound fair to me when I say it.

But I'll give you the answer to my question: They don't want to move away from C/C++, because they don't see a problem. The C/C++ devs I've chatted with on this topic have mostly been convinced they know how to write secure code, that they are capable of taming the distinctive complexity that spirals out of an expanding C++ codebase. Some of them might even be right! They wear their capabilities with pride. To admit that their hard-earned skills were invested in a tool that bears replacing is not something they can consider.

To change, first you have to admit you have a problem. And at least among the C/C++ devs I've talked to about it, few have professed a problem. Combined with the first mover advantage, I fear we'll be having this same debate 50 years from now.

Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:30AM (#29415281)
Agree there is a problem. Now step up to the plate, and fix it.
Re:This shouldn't have happened: A vulnerability postmortem by kovac on Thursday December 02, 2021 @01:36AM (#29413194)
I like this man. I think the only thing we'll look back on is, considering how those who came before us delivered so much with so little (compute power, parallism, safety, tooling), how we fell so far from grace and fuck things right into the ground with slow, unsafe, useless piles of garbage that don't even perform their primary duties well other than collect data.
Re:This shouldn't have happened: A vulnerability postmortem by bottled_poe on Thursday December 02, 2021 @02:35AM (#29413501)
This is great. We give too much merit to armchair experts instead of to people who are naturally too busy making shit happen.
Re:This shouldn't have happened: A vulnerability postmortem by KronisLV on Thursday December 02, 2021 @04:17AM (#29414059)
> C/C++ people may not always be the sharpest tools in the shed, and they may be a dime a dozen.

I feel like you're not entirely serious about it, but i'm not sure about the premise of that statement.

To me, it feels like C/C++ has a barrier of entry that's significantly higher than that of other languages: e.g. everything from JavaScript and Python to those like .NET and Java.

Wouldn't that mean that it'd be easier to learn and be productive in the latter and therefore those devs would be more abundant? Whereas to be capable at C++, you'd need a bit more discipline/patience/whatever and therefore there would be fewer developers?

I see the same in regards to Rust - many talk about it, those who are used to low level development often look into it or adopt it, however those that are used to the languages with a higher level of abstraction, don't venture into actually using it quite as often.

For example, in the JetBrains survey of 2021, about 6% of people used Rust in the past 12 months, but 7% of people are planning of adopting it: https://www.jetbrains.com/lp/devecosystem-2021/ Contrast that with C#, which was used by 21% and had the interest of 4% and Java, which was used by 49% and had the interest of 4%. There are probably better data points about this, i just found this vaguely interesting in regards to that argument. Of course, a counterpoint could be that Rust is new, but if it's so popular, then why people aren't adopting it more quickly?

Summary: regardless of the language being discussed (C/C++/Rust/...), i feel that there will always be fewer developers in the languages with lower level of abstraction, since they feel inherently harder.

Re:This shouldn't have happened: A vulnerability postmortem by davedx on Thursday December 02, 2021 @04:21AM (#29414082)
> To me, it feels like C/C++ has a barrier of entry that's significantly higher than that of other languages: e.g. everything from JavaScript and Python to those like .NET and Java.

Ex C/C++ dev here. I deliberately moved away from working with them in my career (first 10 or so years doing it, last 10 years doing anything but). Not just because "C++ is hard" but because "C++ is intrinsically unsafe in multiple, treacherous ways".

I still have tons of respect for programmers who choose to stay working with C++, but I decided long ago it wasn't for me.

Re:This shouldn't have happened: A vulnerability postmortem by flohofwoe on Thursday December 02, 2021 @05:15AM (#29414412)
> C/C++ has a barrier of entry that's significantly higher than that of other languages

Especially in this context it's important to mention that C and C++ are actually very different languages ;)

Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:46AM (#29415419)
No, its actually absolutely irrelevant to the argument.
Re:This shouldn't have happened: A vulnerability postmortem by dagw on Thursday December 02, 2021 @04:28AM (#29414134)
but i'm not sure about the premise of that statement.

I think it might be a generational thing. I know a lot of 'mediocre' programmers in their 40s and 50s who learnt C++ as their first language and never bothered to really learn anything else (possibly some C#). I'm sure none of them have heard of the JetBrains Survey.

Re:This shouldn't have happened: A vulnerability postmortem by titzer on Thursday December 02, 2021 @06:50AM (#29415001)
> We will understand that those people are the ONLY ones that stepped up to the task over 50 years to write this kind of software, organize standards bodies for their languages and platforms, get their software packaged as part of mainstream operating systems and out into the world, deal with patches from users, and help millions of other people make a living, enable the internet to happen, etc.

While my comment was written to be respectful, yours clearly wasn't. I'll politely point you Niklaus Wirth's work on Oberon as just one example of a giant blind spot in the popular consciousness. Once you learn a bit more about the many, many other systems that were built over the years, you can stop spreading a false narrative of heroism and supremacy.

> The answer will be: these people aren't doers - they are talkers.

Ok, now I am thoroughly done with your comment. Niklaus Wirth was anything but a "talker". Look what was built with Oberon and tell us more about how he was a "talker".

It's OK to not know things. But striking out in profound ignorance is completely unnecessary.

Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:32AM (#29415295)
This comment misses the point so far as to be completely unnecessary.
Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @08:22AM (#29415757)
It pretty much addresses your core thesis and shows it to be based on a fictional retelling of history.
Re:This shouldn't have happened: A vulnerability postmortem by titzer on Thursday December 02, 2021 @08:54AM (#29416094)
Which in itself was a distraction from my core point that C is a hazardous material. Instead of having that conversation, they wanted to drag it into mudslinging, making it about good (heroic! doer) and bad (lazy! talker) people, and profoundly misunderstood history.

We don't vilify people who invented asbestos. We don't vilify workers who installed asbestos. They didn't know better. When we did start to know better, there was a period of denial and even willful ignorance, but eventually the evidence became overwhelming. But now, we know about asbestos, and we're stuck with the consequences to exactly the extent we don't engage in careful cleanups. With the dangers of profoundly unsafe code, we haven't even gotten everyone on the same page yet. It's a long process.

As far talking, I had hoped we could avoid the immature shouty part, as I explicitly acknowledged the many people who put in a lot of work to make C/C++ do things and run fast.

For the doing, I guess I'll be getting back to that. It's far more rewarding than these discussions, frankly.

Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @08:59AM (#29416143)
Frankly it's a bit hard to untangle your point from the way you write. Something to consider.

> and profoundly misunderstood history.

There seems to be a good deal of that.

> They didn't know better.

For what it's worth, we always knew better with C and C++. As I mentioned elsewhere languages were often designed to be safer but as C became more popular the design mistakes of C were pushed on other languages. You can look at FORTRAN, Algol, etc, to see that compiler writers took measures to ensure certain bug classes weren't possible, and those languages were overtaken by those who wanted their code to just compile, correct or not.

Your post definitely comes across as very preachy, so it's odd that you say you were trying to "avoid the immature shouty part" - that's what the parent was basically criticizing you for.

And then to end it on the implication, once again, that some people are "doers" while the rest of us are just posting on HN (the same thing you are doing????) kinda exemplifies it.

Re:This shouldn't have happened: A vulnerability postmortem by native_samples on Thursday December 02, 2021 @05:42AM (#29414573)
I don't think that's actually true. I think it's rather than the market rewards performance above security.

Let's see: is there a mature, maintained TLS stack written in a memory-safe, high performant, cross platform language that Mozilla could have been using for free, instead of NSS? Your argument is that there isn't one because only C++ coders are "doers" instead of "talkers" but this argument is wrong because such a stack does exist. It's JCA/JSSE and ships with every OpenJDK.

JCA/JSSE is a full, open source TLS and cryptography stack. It's written, even at the lowest levels, in Java, in which this kind of memory error cannot occur. It's usable from C/C++ via JNI. It's commercially maintained and has been maintained for decades. It keeps up with the latest features. Its performance in the latest versions is comparable to OpenSSL even (not always in the past).

Mozilla could have used it. They could have improved it, even. They've shipped browsers with integrated Java before, in times when computers were much less powerful. They chose not to, why, well, the charitable explanation might be download size or performance, but we could just as easily argue it's because C++ guys don't really care about security. Bugs happen, they ship a fix, people let them off the hook. Eh another buffer overflow, who cares. Not like anyone is going to switch products over that, right?

Reality is, our industry could rewrite lots of software in safer languages and already has. Java has tons of libraries that do the same thing as C++ libraries, and which could be used today. You can shrink HotSpot down to about 7mb compressed, last time I experimented with this, which is like 10% of the size of a typical mobile app. It's just not a big deal. Perhaps the biggest problem is that JNI is an awkward FFI but there are fixes for that also, the new FFI (Panama) is a lot better than the old one.

Re:This shouldn't have happened: A vulnerability postmortem by scoutt on Thursday December 02, 2021 @09:45AM (#29416737)
Sincerely, as a C/C++ programmer, if I have to start a new project and you propose to link a Java library, it'd give me the creeps. Let's say I agree: does my project now requires to install a Java VM together with my project? Which Java VM? I still don't understand the difference between OpenJDK and the "other one"... JDK? JRE? I only installed it once many years ago because Eclipse CDT required it and I might have wrote "sudo update-java-alternatives --set /path/to/java/version" one too many times (AOSP development).

OK. I'll make an installer to install Java with my project. But the customer has another older version, or a competitor version, or ..... I don't care anymore.

Do I have to build this library? What paraphernalia do I have to install and learn to automatize, build and test a complex Java library? Do I have to maintain a JNI interface too?

Where is the JCA/JSSE source code? (I can't find it for real) And who maintains it? Oracle? What if Oracle one day pulls the plug or comes after me with one of their famous legal moves just because?

These are my concerns. You might try to convince me, but I already learnt about C/C++ libraries, DLL hell, etc, and have all my fixes in place.

> Bugs happen

They will still happen. Put your heart at peace.

Re:This shouldn't have happened: A vulnerability postmortem by native_samples on Thursday December 02, 2021 @11:03AM (#29417855)
I think you're making my point for me, no? Your arguments are all variants on "I don't understand modern Java and prefer to keep writing buffer overflows than finding out".

To answer your questions:

1. Java is these days like UNIX, there are lots of "distros". OpenJDK is the upstream on which most of them are based. You can just use that unless you have some particular preference for the other vendors. However the compatibility situation is much better, there aren't any compatibility issues and JVMs are all more or less drop-in replacements for each other (Android is an exception but has got a lot better over time).

2. You don't need to install Java alongside your app. The JVMs is just a library and a few data files. You can easily bundle it with your app and the user will never know.

3. JCA/JSSE is built in to Java so there's nothing to build. Additionally you don't have to build Java libraries to use them anyway, because binary distribution works so everyone just distributes binary JAR files.

4. The source code is spread around several modules because JCA is pluggable. But for example a lot of the crypto code is found here:

https://github.com/openjdk/jdk/tree/master/src/java.base/sha...

5. Oracle maintains it but other companies contribute, like Amazon, Microsoft, Red Hat etc (well, to Java as a whole). It's released under the GPL with Classpath exception license. They aren't going to come after you for using it - there are 12 million Java developers in the world. The only company that has ever got sued is Google and that's because they violated the (at the time non open source) license of Java to make a mobile version that wasn't from Sun. They knew they were doing it and chose to chance it anyway (which worked out quite well for them actually). Since then Java became fully open source and this no longer applies, and it'd have never applied to normal users of it anyway.

I'm not trying to convince you here of anything, only to point out that everyone saying "there was no alternative" is just wrong. The failure mode here is not sufficiently advanced fuzzers. The failure is that there has been an alternative to NSS for years but C++ shops like Mozilla keep maintaining crappy decades old C libraries with memcpy calls all over the place, because they know and like C. That's it. That's all it boils down to.

Re:This shouldn't have happened: A vulnerability postmortem by scoutt on Thursday December 02, 2021 @11:45AM (#29418553)
> "I don't understand modern Java"

Exactly!

> "and prefer to keep writing buffer overflows than finding out"

Not exactly. I would rather take the risk of writing buffer overflows than using over-bloated infrastructure, being in code size and/or performance, tooling, including learning time. It's a tradeoff. It's always a tradeoff. Mozilla might have had their own reasons, and probably not on a whim. If I have to write the software for an ECU the approach will be different (no, it won't be Java).

I've been in the industry for a long time now (C/C++, mostly embedded). The pattern is the same. This time is you with Java, it might be the Rust/Go crowd , the other 4 users with Ada or the 2 users with FP: everybody loves to hate C/C++ and everybody have their whiteboard reasons, but C/C++ has been pushing the world forward for the last 30 years minimum, and it's not stopping anytime soon. There must be a reason other than "C/C++ programmer are lazy dummies, they prefer to write bugs instead of learning the marvels of [insert random language here], because they are in love with memcpy", don't you think?

The code is there. You can try to link Java JCA/JSSE to Firefox or whatever the project is about. I'm interested in learning how it looks and how it works.

Re:This shouldn't have happened: A vulnerability postmortem by native_samples on Thursday December 02, 2021 @12:42PM (#29419541)
Right, as I said up thread: "I think it's rather than the market rewards performance above security." - so we're not really disagreeing.

In this case I doubt there'd be a big CPU overhead. There'd be some memory overhead, and browsers compete on that, although of course they're kind of memory pigs already.

"The code is there. You can try to link Java JCA/JSSE to Firefox or whatever the project is about. I'm interested in learning how it looks and how it works."

I think once Panama (the new FFI) gets released it'd be interesting to experiment with this sort of thing, though I don't care about Firefox personally. That would eliminate a lot of the tedious boilerplate you'd otherwise have to churn out.

Re:This shouldn't have happened: A vulnerability postmortem by badpun on Thursday December 02, 2021 @11:10AM (#29417972)
That’s the reason Java is used almost exclusively on servers. None of your pain points (well maybe except Oracle getting litigious) apply there.
Re:This shouldn't have happened: A vulnerability postmortem by ksec on Thursday December 02, 2021 @08:53AM (#29416076)
>The answer will be: these people aren't doers - they are talkers.

I think it is the market values more features and performance, with incremental improvement to current technology at small expense of security. Or the cost of features and performance with currently available tools and work force ( C / C++ ) is vastly cheaper than rewriting it in what ever PL that is.

The only reason why we are starting to do it now, is because the total computer market is 10x larger than we had. Tech is also used 10x more and all of a sudden the cost of security may be worth the cost to switch.

It is all a economic model. or Market Interest. Nothing to do with Armchair and Doers.

Re:This shouldn't have happened: A vulnerability postmortem by renewiltord on Thursday December 02, 2021 @03:40AM (#29413842)
Lmao. Absolute perfection. Software that exists has way more value.
Re:This shouldn't have happened: A vulnerability postmortem by oblio on Thursday December 02, 2021 @05:04AM (#29414358)
Not when that software and the ecosystem around it, which are entrenched, actively fight against change.

If you think that's not true, C++ programmers, raise your hand if you were ever working on/contributing to a C program and you proposed C++ as a safer/better language and got shot down.

So even C++ is facing the same kind of challenges now faced by newer programming languages.

Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:35AM (#29415330)
> So even C++ is facing the same kind of challenges now faced by newer programming languages.

Couldn't agree more with this.

Re:This shouldn't have happened: A vulnerability postmortem by Chris2048 on Thursday December 02, 2021 @05:41AM (#29414566)
> wonder why all the Lisp and Haskell programmers .. didn't get off their asses

I'm sure all these things exist in these languages, totally unused.

Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:16AM (#29415178)
I actually agree with you - in many cases, I am sure they do.

But are they complete, and usuable enough in production? Did they ever acquire enough mindshare?

Did they get enough buy in?

The clear answer is no.

Re:This shouldn't have happened: A vulnerability postmortem by Chris2048 on Thursday December 02, 2021 @07:25AM (#29415238)
Mindshare and buy-in might be an issue of the ecosystem though. Not enough lisp/Haskell devs.
Re:This shouldn't have happened: A vulnerability postmortem by nineteen999 on Thursday December 02, 2021 @07:34AM (#29415320)
Maybe thats because while those languages solve particular problem, they, or their proponents aren't solving the particular problems at hand.
Re:This shouldn't have happened: A vulnerability postmortem by Chris2048 on Thursday December 02, 2021 @09:30AM (#29416535)
Perhaps, but that's speculation. If nor corp will invest outside fortran, cobol, C++ or java then there's a limit to how much problem solving a small community can do on its own dime.

The point is, I think speculative characterisations of <community>-dev members as lazy/unproductive is off mark.

Re:This shouldn't have happened: A vulnerability postmortem by kortilla on Friday December 03, 2021 @05:37AM (#29428541)
There are likely an order of magnitude more lines of Java, Python, and JavaScript out there that don’t have this problem. You’ve painted a pretty dumb false dichotomy here with the Haskell vs C trope.
Re:This shouldn't have happened: A vulnerability postmortem by saurik on Thursday December 02, 2021 @12:06AM (#29412738)
More strongly: the idea that "fuzzing" is "doing all the right things" is insane and disappointing of a narrative. The code snippets I am seeing here are ridiculous with manual, flag-based error handling? There is a use of something like memcpy that isn't hidden behind some range abstraction that manages all of the bounds?! This code probably is riddled with bugs, because they are doing all the wrong things, and fuzzing the result counts for nothing. C++ sucks, but it is way better than this.
Re:This shouldn't have happened: A vulnerability postmortem by ehsankia on Thursday December 02, 2021 @02:47AM (#29413562)
I don't think anyone is under the illusion that "fuzzing is doing all the things" or that it is a replacement for actual unit, integration and E2E tests.

It still is true that fuzzing has managed to find a whole slew of bugs that were otherwise not found, and it is generally easier to automate and add without requiring a ton of engineer time. It is meant to be an addition on top, not a replacement for other techniques.

Re:This shouldn't have happened: A vulnerability postmortem by lrem on Thursday December 02, 2021 @05:01AM (#29414343)
"All the right things" included, according to the actual narrative in the post:

- Extensive non-fuzzing testing.

- Regular static analysis.

- Being actively maintained and overseen by a competent security team, with a bug bounty program on top.

I believe I've also seen mentions of independent security audits outside the post.

Edit: emphasis is on the fuzzing, because that is how the bug was finally discovered.

Re:This shouldn't have happened: A vulnerability postmortem by Woodi on Thursday December 02, 2021 @01:13AM (#29413078)
> but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.

Maybe. But there is good reason for C dominance - it's low level and close to systems as in "syscalls and stuff". And we need that level of systems control, not only for max performance but also for not loosing what is available in hw and os.

Asm is the other option to have full functionality :) Maybe it's just case of available libraries but still C is curently only one option for feature completness. And on low levels all you have is bytes and bits and calls and that allows to do everything with bits, bytes and calls - Core Wars style - and most of languages try to prevent or easy use of that by cutting possibilities, eg. "goto is so nono !11".

And yes, C is not perfect and bullshit even grows, eg. in last years NULL checks are "optimized-out"...

C actually should be improved and not fucked up into forced disuse...

Re:This shouldn't have happened: A vulnerability postmortem by oblio on Thursday December 02, 2021 @05:05AM (#29414364)
> C actually should be improved and not fucked up into forced disuse...

It can't, it won't, it isn't.

The last major update has C99 and even that was minor from a security point of view.

For better or worse, it's frozen.

Re:This shouldn't have happened: A vulnerability postmortem by native_samples on Thursday December 02, 2021 @05:44AM (#29414583)
C isn't dominant in a lot of areas of software, obviously.

As for NSS. It's a pure data manipulation library. Algorithms, data structures, not more. It doesn't need to do syscalls and stuff. There's not much benefit to writing it in C and as we just saw, a lot of downsides. It could have been written in many other languages.

Re:This shouldn't have happened: A vulnerability postmortem by pas on Thursday December 02, 2021 @04:55PM (#29423259)
Memory is cheap. (But not free of course!) Scala, Kotlin, and Java exist. The HotSpot VM is amazing, its JIT is mindblowing. (Probaly even TS + V8 would do the job for high level stuff.)

It's complete nonsense that C is needed for accessing the HW. High level languages can do low-level bibanging just fine.

So yes, there's a good reason, but that is inertia. Literally too big to fail, so it keeps on living. Even if we tomorrow outlawed C, drafted people to start rewriting things in a memory-managed language, and so on, it would take decades to get rid of it.

Re:This shouldn't have happened: A vulnerability postmortem by lenkite on Thursday December 02, 2021 @01:09AM (#29413056)
Is it OK to remove modern C++ from your statement ? Using a `std::vector<std::byte>` wouldn't cause this problem. Don't know why everyone always berates C++ for vulnerabilities in traditional C code.
Re:This shouldn't have happened: A vulnerability postmortem by sebcat on Thursday December 02, 2021 @02:05AM (#29413365)
It wouldn't cause the problem in itself perhaps, but I find it a bit reductive to look at the type in isolation like that. Sometime, somewhere, someone will call std::vector<T,Allocator>::data on that vector and use the resulting pointer as the src argument to memcpy (or some other function), and someone else will make a change that causes an overflow of the dst buffer. Shit happens and code written in modern C++ also has bugs and some of those bugs have security implications.
Re:This shouldn't have happened: A vulnerability postmortem by Animats on Thursday December 02, 2021 @02:19AM (#29413427)
Sometime, somewhere, someone will call ...

I call this the mold seeping through the wallpaper. C++ tries to paper over C's terrible array model by using collection class templates, but those constructs leak. Too many things need raw pointers.

Re:This shouldn't have happened: A vulnerability postmortem by snovv_crash on Thursday December 02, 2021 @02:23AM (#29413449)
You're just talking about more C interfaces, not C++. The same thing would happen in Rust if you tried to pass a chunk of memory to C.
Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @09:16AM (#29416320)
> The same thing would happen in Rust if you tried to pass a chunk of memory to C.

In an 'unsafe' block.

Re:This shouldn't have happened: A vulnerability postmortem by snovv_crash on Thursday December 02, 2021 @10:54AM (#29417684)
The memory corruption would happen in the C code though. By the time that the program is actually affected by the memory error it could be much later, back in Rust code, and now you don't have any tools for debugging memory corruption because "that never happens in Rust".
Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @11:02AM (#29417845)
The very first thing you would do is audit for 'unsafe' though.
Re:This shouldn't have happened: A vulnerability postmortem by snovv_crash on Friday December 03, 2021 @06:16AM (#29428762)
Assuming you ever encounter the issue, sure. But this bug was only triggered by long keys, which are outside of the normal operating paradigms. So your Rust code calling into a C API would have had exactly the same security vulnerability as C++.

On the other hand, something like Java via JNI wouldn't, because it copies the data to a different address space as it goes through the language boundary. Horribly inefficient, but at least the C code only causes security issues in the C regions. By the time it gets back into Java it either crashes or it's safe again, no undefined behaviour leaks through the API boundary.

Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Friday December 03, 2021 @12:25PM (#29432540)
Copying the data before the FFI boundary would have done absolutely nothing to prevent this bug or make it any less difficult to exploit tho
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @02:54AM (#29413597)
True, except the code will be tainted and it is easy to find it.
Re:This shouldn't have happened: A vulnerability postmortem by snovv_crash on Thursday December 02, 2021 @10:56AM (#29417727)
Memory corruption doesn't always trigger segfaults. I don't believe it will be obvious why some random other part of your program will start giving intermittent errors even if it is in Rust.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @11:09AM (#29417952)
In Rust, or any other systems programming language with unsafe code blocks, all the way back to JOVIAL and ESPOL, one can search for those code blocks.

At very least they provide an initial searching point.

On C, C++ and Objective-C, any line of code is a possible cause for memory corruption, integer overflow, or implicit conversions that lead to data loss.

Re:This shouldn't have happened: A vulnerability postmortem by snovv_crash on Thursday December 02, 2021 @12:12PM (#29419030)
This is starting from the point of knowing it is a memory corruption issue though. From my experience, memory corruption usually manifests in logic or data behaviour changes. In a Rust program you'd probably spend a few days pulling your hair out trying to understand why some data structure doesn't do what it's supposed to before considering that it's one of the unsafe blocks.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Friday December 03, 2021 @02:36AM (#29427581)
Yeah, but at least you know where to start searching afterwards.

This applies to other languages with unsafe code blocks, note that JOVIAL and ESPOL were the first ones offering such capability.

Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @02:53AM (#29413590)
It would, because ISO std::vector<std::byte> doesn't do bounds checking by default unless you turn on the compiler checks in release builds, use at() everywhere, or create you own `std::checked_vector<std::byte>`.
Re:This shouldn't have happened: A vulnerability postmortem by CJefferson on Thursday December 02, 2021 @04:56AM (#29414313)
The default [] for vector doesn't do bound checking, so I don't feel it helps they much.

One thing in practice I like about Rust over C++ is the "easy option" is safe, and you have to do things like write 'unsafe' to opt out of safety.

Certainly when teaching C++ I wish the defaults were safer. I can teach ".at", but it isn't what you see in most code / tutorials.

Re:This shouldn't have happened: A vulnerability postmortem by gpderetta on Thursday December 02, 2021 @08:19AM (#29415721)
-D_GLIBCXX_ASSERTIONS
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @08:38AM (#29415923)
Good luck advocating for that on release code.
Re:This shouldn't have happened: A vulnerability postmortem by flohofwoe on Thursday December 02, 2021 @05:18AM (#29414436)
People who memcpy bytes into a destination without caring if there's enough room would also not hesitate to use memcpy to copy bytes into a std::vector though.
Re:This shouldn't have happened: A vulnerability postmortem by lenkite on Saturday December 04, 2021 @04:00PM (#29444269)
Re:This shouldn't have happened: A vulnerability postmortem by raverbashing on Thursday December 02, 2021 @02:33AM (#29413484)
> Using a `std::vector<std::byte>`

And herein lies the problem. Your statement is correct. But that type definition is exactly why C++ feels like a 50's car retrofitted with an electric drivetrain and a DVD player on the dash

It will almost run like a modern thing but it will fail when it shouldn't.

Re:This shouldn't have happened: A vulnerability postmortem by speed_spread on Wednesday December 01, 2021 @11:08PM (#29412466)
At least Asbestos is inert once in place.
Re:This shouldn't have happened: A vulnerability postmortem by masklinn on Thursday December 02, 2021 @12:53AM (#29412983)
Isn’t asbestos always inert unless you matchmake it to a ridiculously strong oxidiser?

In my recollection it dealt purely physical damage, and its inability to react with much of anything is what leads to its accumulation in dwellings.

Re:This shouldn't have happened: A vulnerability postmortem by afiori on Thursday December 02, 2021 @02:14AM (#29413403)
I believe they mean inert as it relate to health, not chemically.
Re:This shouldn't have happened: A vulnerability postmortem by adrianN on Wednesday December 01, 2021 @11:46PM (#29412661)
Unfortunately it's too much work to throw away all code written in unsafe languages. So it's valuable to try and improve tools and techniques that make these languages less unsafe.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @08:40AM (#29415937)
One reason why we are getting hardware memory tagging to transform modern computers into basically C Machines, is that everyone kind of gave up doing it at sofware level in regards to C and C++.
Re:This shouldn't have happened: A vulnerability postmortem by titzer on Thursday December 02, 2021 @08:57AM (#29416120)
Just compile it all to Wasm and we'll run it in virtualization.

Give us memory segments back and I think we have a shot of making this a reality.

Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @11:03AM (#29417865)
Except WASM made the big mistake of not having bounds checking on linear memory, so attacks with input data that corrupt internal state and by it try to influence the outcome of WASM modules behaviour are still a good attack vector.

Being sandboxed is good, however if one can start an internal fire from the outside, not so much.

Re:This shouldn't have happened: A vulnerability postmortem by titzer on Thursday December 02, 2021 @03:57PM (#29422438)
Not sure what you were expecting, but no, Wasm doesn't magically make memory-unsafe languages internally uncorruptible, it limits all corruption to internal state. If you look at the PL research stretching back a couple decades on how to do that for C, you are looking at integer factor performance overheads in the general case. Wasm also doesn't make anything less safe than it was before (ok, ok, modulo the current lack of read-protected memory), and since we wisely chose to make the execution stack non-addressable, has CFI by default.

Wasm's sandboxing makes it impossible to escalate the privilege of a program or acquire capabilities that it didn't have before (i.e. were not explicit imported and granted). That's a strictly stronger primitive than a random userspace Unix process.

Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Friday December 03, 2021 @01:41AM (#29427294)
I was expecting that people don't oversell WebAssbemly as some kind of magic pixie dust of security, anyone that knows a bit about security spots those flaws.

You know what is also sandboxed? An OS process.

Ah, but an OS process has a wider syscalls surface, well lets than bring WASI or JavaScript bindings into the picture.

Re:This shouldn't have happened: A vulnerability postmortem by titzer on Saturday December 04, 2021 @11:48AM (#29442023)
At least we are on the same page on the level of sandboxing provided. For the record, it was your comment that called it a "big mistake" to not have finer-grained memory protection. I explained why we did that. And now you're saying someone "oversold" it as magic pixie dust? I'm not sure which hype train you're referring to, TBH. It wasn't core members or the CG, as we understood very clearly which security properties we were gaining and which we were not gaining.
Re:This shouldn't have happened: A vulnerability postmortem by pjmlp on Thursday December 02, 2021 @02:48AM (#29413568)
One of Bjarne's talks at CppCon 2021, is yet advocating again for the Code Guidelines, because they just keep being ignored by the community at large.
Re:This shouldn't have happened: A vulnerability postmortem by wheelerof4te on Thursday December 02, 2021 @10:43AM (#29417508)
C ABI is still stable after all these years. Programs written in "The C Programming Language" bible still compile and work fine.

That's why.

Re:This shouldn't have happened: A vulnerability postmortem by gotbeans on Thursday December 02, 2021 @04:16AM (#29414050)
Imagine maintaining a long-term project in the first place.

Such crazyness, like why?

Re:This shouldn't have happened: A vulnerability postmortem by GoblinSlayer on Thursday December 02, 2021 @04:05AM (#29413998)
Ironically STL supports bound checking, but is always turned off.
Re:This shouldn't have happened: A vulnerability postmortem by rfoo on Wednesday December 01, 2021 @03:16PM (#29408618)
What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs.

Otherwise, the same story could be told as a generic software testing joke: "unit-tests are short-sighted and coverage lies", i.e. an "extremely well-maintained codebase, with extensive unittest, >98% test coverage and constantly scanned by all-static-analyzers-you-may-come-up" can have fatal, trivial bugs.

Re:This shouldn't have happened: A vulnerability postmortem by bluejekyll on Wednesday December 01, 2021 @03:59PM (#29409056)
Ah, brings to mind one of my favorite Dijsktra quotes, "Program testing can be used to show the presence of bugs, but never to show their absence!"

I've never understood that to mean that he wasn't in favor of automated testing, only that it's got its limits. In this case, they now know a test case that was missing.

Re:This shouldn't have happened: A vulnerability postmortem by bialpio on Wednesday December 01, 2021 @07:27PM (#29411010)
Yup, in my previous gig we had an approach that if there is a bug in the code that the tests didn't catch, it's actually 2 bugs: a product and a test issue. Similarly, if a dev needed to have access to the live debugging session to diagnose the issue, it meant there was yet another bug (to improve debuggability of the product). It was quite nice, taught me a lot.
Re:This shouldn't have happened: A vulnerability postmortem by davedx on Thursday December 02, 2021 @04:28AM (#29414137)
Indeed. This is why I find 100% unit test coverage can actually be harmful: it gives you an illusion of safety.
Re:This shouldn't have happened: A vulnerability postmortem by lucb1e on Wednesday December 01, 2021 @03:22PM (#29408680)
> What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs.

Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buffer overflows are security issues. (Presuming no mitigations, of course, since those are separate from the bug itself.)

Re:This shouldn't have happened: A vulnerability postmortem by PeterisP on Wednesday December 01, 2021 @04:20PM (#29409274)
As a crude example, there sometimes are off-by-one bugs which allow the buffer to overflow by one byte, and that single overflowing byte is always 0 (the last bye in a zero-terminated string), and it overwrites data in a variable that doesn't affect anything meaningful, giving you a buffer overflow with no security impact.
Re:This shouldn't have happened: A vulnerability postmortem by trissylegs on Thursday December 02, 2021 @01:14AM (#29413082)
Although single zero byte overflows are sometimes exploitable.

https://googleprojectzero.blogspot.com/2014/08/the-poisoned-...

Re:This shouldn't have happened: A vulnerability postmortem by lelanthran on Thursday December 02, 2021 @03:24AM (#29413752)
> Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given.

Not necessarily:

1. Most compilers use aligned variables and fields unless forced not to with a flag. Going 3 bytes over an array of 12 bytes can result in an overflow that is never detectable at runtime because the extra memory being used is used exclusively by the bug.

2. Malloced memory is harder (but not impossible) to turn into an unintended execution because the pages may be marked for data only by the OS. The stack pages are ~marked~ [EDIT: ~NOT marked~] as executable.

There's probably millions of buffer overflows that will not only never be exploited, but will also never be detected.

Re:This shouldn't have happened: A vulnerability postmortem by rfoo on Wednesday December 01, 2021 @03:28PM (#29408747)
Sorry, I mean the special part is "the bug itself is a memory corruption". The second sentence is a quick explanation for those not in our bubble.
Re:This shouldn't have happened: A vulnerability postmortem by staticassertion on Thursday December 02, 2021 @09:20AM (#29416392)
Lots of bugs aren't particularly exploitable. As Tavis notes, one of the major problems here is that there's a structure containing function pointers adjacent to the buffer.

If all you have is a stack overflow you may have your work cut out for you.

Further, while this bug may exist, assertions elsewhere could have meant it was unexploitable. So in isolation it's an overflow but in context it's not reachable. That didn't happen here. This happens a lot actually.

Re:This shouldn't have happened: A vulnerability postmortem by Lhiw on Wednesday December 01, 2021 @03:28PM (#29408739)
Unit tests aren't really for bug catching, they're to ensure you haven't changed behavior when you don't expect to.

They enable refactoring code in ways not possible without them.

Re:This shouldn't have happened: A vulnerability postmortem by moring on Thursday December 02, 2021 @12:50AM (#29412968)
FWIW, this does not match my experience. I have caught lots of bugs with unit tests, especially in code that is fundamentally complex (because it does complex things, not because it needs polishing). OTOH, refactorings often span units because real simplification comes from changing the ways units interact, or even which units exist, so the tests have to be changed anyway.

Granted, even tests that have to be changed have some value in securing a refactoring.

Re:This shouldn't have happened: A vulnerability postmortem by rhdunn on Thursday December 02, 2021 @03:07AM (#29413668)
If you are writing tests to check new code in tandem with writing that code (either via TDD, or some other code-test loop), or are writing tests for existing code, you can (and usually will) find and fix bugs. Likewise if you are investigating a problem and write one or more test cases to check the behaviour.

Once those tests have been written, then they act as regression tests like the parent comment notes.

On "unit tests", I view the behaviour of the class/function and all its dependencies as a single unit for the purpose of testing. I've never liked the idea of needing to mock out a class in order to test another class, just because the class being tested makes use of it. The only case where mocks/stubs/etc. should be used is when interacting with an external component like a database or HTTP API. -- You don't see people that do this mocking out a list, string or other library classes, so why should project classes be any different.

Re:This shouldn't have happened: A vulnerability postmortem by moring on Thursday December 02, 2021 @08:08AM (#29415618)
To clarify, when I wrote "the ways units interact", I was referring to units that represent external dependencies in some kind (often a database, as you said). Many refactorings change those interactions in some way.

I agree that there is no reason to mock data containers or other classes with fully self-contained behaviour.

Re:This shouldn't have happened: A vulnerability postmortem by jandrese on Wednesday December 01, 2021 @02:48PM (#29408278)
> - "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes"

This is the one that seemed short sighted to me. It's a completely arbitrary (and small!) limit that blinded the fuzzer to this very modest sized buffer overflow.

Re:This shouldn't have happened: A vulnerability postmortem by jjoonathan on Wednesday December 01, 2021 @03:06PM (#29408497)
Oh, the fools! If only they'd built it with 6001 hulls! When will they learn?
Re:This shouldn't have happened: A vulnerability postmortem by 6chars on Wednesday December 01, 2021 @03:20PM (#29408658)
Thank you! So many hindsight fortune tellers here.
Re:This shouldn't have happened: A vulnerability postmortem by l33t2328 on Thursday December 02, 2021 @09:30AM (#29416532)
Not using small bounds on your fuzzing size isn’t exactly something you can only know via hindsight.
Re:This shouldn't have happened: A vulnerability postmortem by Terry_Roll on Wednesday December 01, 2021 @09:13PM (#29411716)
Predictable Human Behaviour, you can literally count on it! LOL

Anyway the post mortem analysis is interesting because it gives away peoples knowledge or lack of.

What is it they say? The Devil is in the detail!

Re:This shouldn't have happened: A vulnerability postmortem by js2 on Wednesday December 01, 2021 @02:55PM (#29408368)
The buffer holds 2K, so this limit alone which exceeds the buffer by 8K'ish didn't blind the fuzzer. It's not clear a larger input would've caught anything due to other "what went wrong" items, specifically "each component is fuzzed independently."
Re:This shouldn't have happened: A vulnerability postmortem by a-priori on Wednesday December 01, 2021 @02:56PM (#29408378)
The problem is that the search space grows (exponentially?) as you increase the fuzzer’s limit. So there’s a cost, and likely diminishing returns, to raising that limit.
Re:This shouldn't have happened: A vulnerability postmortem by UncleMeat on Wednesday December 01, 2021 @03:43PM (#29408894)
Coverage-guided fuzzing dramatically mitigates the exponential nature of the search space. It used to be that searching for magic bits was impossible with fuzzing but now it is nearly trivial.
Re:This shouldn't have happened: A vulnerability postmortem by jandrese on Wednesday December 01, 2021 @03:04PM (#29408480)
Are they checking every possible overflow up to the max? Like no overflow at 7377 bytes, lets try 7378...

While I can see targeting near natural boundaries (1025 bytes for example), you should be able to skip over most of the search space and verify that it doesn't blow up on enormous values like 16777216 bytes.

Re:This shouldn't have happened: A vulnerability postmortem by lucb1e on Wednesday December 01, 2021 @03:40PM (#29408865)
It's not that simple though. Single-variable integer overflows can be checked like that, but when the critical byte in a buffer might be at positions 1 through {bufferlength}, you have to do a shotgun approach and see if anything sticks, and at some point the number of possible combinations grows too big even for that.

I'm not an expert on fuzzing myself, but generally I do see the point of having a limit here. Why, then, that limit was not chosen to be the max size for each of the length-checked inputs, I don't know. That does seem a bit more obvious, but also I just read this article so I can't prove that I wouldn't have made the same mistake.

Re:This shouldn't have happened: A vulnerability postmortem by moring on Thursday December 02, 2021 @12:54AM (#29412991)
Is it possible to feed a fuzzer with information from static analysis to limit the search space? Such as, if you have a check like "someParameter > 0" in the code, have the fuzzer generate a positive, negative, and zero value for someParameter, but not thousands of them -- at least not based on this check alone -- because they will all behave the same.
Re:This shouldn't have happened: A vulnerability postmortem by baby on Thursday December 02, 2021 @01:46AM (#29413247)
There's whitebox fuzzing that's starting to become a thing, when your fuzzer gets stuck you give the non-giving corpus to an SMT solver and it'll try to see if there exists an input that could uncover a new path. I'm really excited about these but haven't really followed the advances.
Re:This shouldn't have happened: A vulnerability postmortem by baby on Thursday December 02, 2021 @01:44AM (#29413239)
If you increase the limit you lose in coverage as you can only do so many exec/s
Re:This shouldn't have happened: A vulnerability postmortem by Veserv on Wednesday December 01, 2021 @04:07PM (#29409140)
How do you know that it was actually "extremely well-maintained"? Everybody thought OpenSSL was well-maintained since it was used as critical infrastructure by multi-billion dollar megacorporations, but it was actually maintained by two full-time employees and a few part-time volunteers with maybe a quick once-over before a commit if they were lucky. How about sudo, a blindly trusted extremely sensitive program [1], which is maintained by basically one person who has nearly 3,000,000(!) changes[2] over 30 or so years?

Assuming that something is well-maintained because it is important is pure wishful thinking. Absent a specific detailed high quality process, or an audit that they conform to a well-established process that has demonstrably produced objectively high-quality output in a large percentage of audited implementations of that process (thus establishing nearly every instance of the audited process -> high quality output) all evidence indicates that you should assume that these code bases are poorly maintained until proven otherwise[3]. And, even the ones that are demonstrably maintained usually use very low quality processes as demonstrated by the fact that almost nobody working on those projects would be comfortable using their processes on safety-critical systems [4] which is the minimum bar for a high quality process (note the bar is "believe it is okay for safety-critical"). In fact, most would be terrified of the thought and comfort themselves knowing that their systems are not being used in safety-critical systems because they are absolutely not taking adequate precautions, which is a completely reasonable and moral choice of action as they are not designing for those requirements, so it is totally reasonably to use different standards on less important things.

[1] https://news.ycombinator.com/item?id=25919235

[2] https://github.com/sudo-project/sudo/graphs/contributors

[3] https://xkcd.com/2347/

[4] https://xkcd.com/2030/

Re:This shouldn't have happened: A vulnerability postmortem by nullityrofl on Wednesday December 01, 2021 @04:15PM (#29409225)
Tavis explains clearly why he thinks it's well maintained in the post, complete with linkages to source code. To paraphrase:

[...] NSS was one of the very first projects included with oss-fuzz [...]

[...] Mozilla has a mature, world-class security team. They pioneered bug bounties, invest in memory safety, fuzzing and test coverage. [... all links to evidence ...]

Did Mozilla have good test coverage for the vulnerable areas? YES.

Did Mozilla/chrome/oss-fuzz have relevant inputs in their fuzz corpus? YES.

Is there a mutator capable of extending ASN1_ITEMs? YES.

I don't think at any point anyone assumed anything.

Re:This shouldn't have happened: A vulnerability postmortem by jcranmer on Wednesday December 01, 2021 @05:24PM (#29409943)
OpenSSL is a project which is treated as a "somebody else's problem" dependency by everybody, and the extent to which everybody cares about TLS support, it's basically an "open up a TLS socket and what do you mean it's more complicated than that" situation.

By contrast, NSS is maintained by Mozilla as part of Firefox, and, furthermore, its level of concern is deep into the "we don't want to enable certain cipher suites, and we have very exacting certificate validation policies that we are part of the effort in defining"--that is to say, NSS isn't a "somebody else's problem" dependency for Mozilla but a very "Mozilla's problem" dependency.

That said, this is CERT_VerifyCertificate, not mozilla::pkix, and since this is not used in Firefox's implementation of certificate validation, I would expect that this particular code in the library would be less well-maintained than other parts. But the whole library itself wouldn't be in the same camp as OpenSSL.

Re:This shouldn't have happened: A vulnerability postmortem by antod on Wednesday December 01, 2021 @05:02PM (#29409705)
> Everybody thought OpenSSL was well-maintained since it was used as critical infrastructure by multi-billion dollar megacorporations

I wasn't under the impression anybody who knew the project ever really thought that. Some other people may have assumed that as a default if they hadn't looked into it.

This article spells out a whole bunch of reasoning why this particular library was well maintained though. There's a difference between reasoning based on evidence and assumptions.

Re:This shouldn't have happened: A vulnerability postmortem by tptacek on Wednesday December 01, 2021 @05:28PM (#29409978)
I don't know anybody who thought OpenSSL was well-maintained in and before the Heartbleed era (it's a fork of SSLeay, which was Eric Young's personal project). Post-Heartbleed --- a decade ago, longer than the time lapse between SSLay and OpenSSL --- maintenance of OpenSSL has improved dramatically.
Re:This shouldn't have happened: A vulnerability postmortem by Godel_unicode on Wednesday December 01, 2021 @11:17PM (#29412520)
I think you and a sibling comment might be too close to the problem. When heartbleed dropped my Twitter feed had a few crypto engineers saying "I mean, eventually this was going to happen" and a ton of developers whose main language starts with a "p" going "how?? OpenSSL is core plumbing of the internet, it can't be this bad can it???".

Edit: to be clear, not maligning the "p" language developers, I was one myself. Simply saying there are a ton of technical people who are blissfully ignorant of the tiny pillars of clay that hold up the internet.

Re:This shouldn't have happened: A vulnerability postmortem by bee_rider on Thursday December 02, 2021 @12:24AM (#29412835)
To be fair, it was unreasonable to expect Prolog coders to keep up with crypto advancements, I mean the language pre-dates SSL by decades.
Re:This shouldn't have happened: A vulnerability postmortem by tannhaeuser on Thursday December 02, 2021 @01:15AM (#29413087)
Care to expand on what Prolog coders have to do with OpenSSL? Not getting the context here.
Re:This shouldn't have happened: A vulnerability postmortem by wolf550e on Thursday December 02, 2021 @02:12AM (#29413389)
A joke about "developers whose main language starts with a "p"", meaning presumably perl, php and python. I guess in this context, ruby and JavaScript are also languages starting with "p".
Re:This shouldn't have happened: A vulnerability postmortem by Godel_unicode on Thursday December 02, 2021 @11:38AM (#29418458)
Don't worry, I told both of them what was going on ;)
Re:This shouldn't have happened: A vulnerability postmortem by seanhunter on Thursday December 02, 2021 @02:38AM (#29413513)
Pascal does predate the era of SSL by some margin to be fair.
Re:This shouldn't have happened: A vulnerability postmortem by hn_go_brrrrr on Thursday December 02, 2021 @01:49AM (#29413266)
Yeah, those Perl hackers are really out of touch :-)