You can't really divorce UNIX from C.
The only other language option for an smtpd server in OpenBSD base would have been Perl, and I bet people would complain about that just the same.
And perhaps we wouldn't be able to review it properly if it was written in Perl. Perhaps Marc Espie could. But OpenBSD has plenty of people on the team who are well trained at auditing C.
I'll be doing my part and read some smtpd code during an 8h train ride tomorrow. I already read one file and found one bug in it and told gilles about it. That's how this works. Changing the language won't magically fix anything.
Wonderful! This is exactly what I (and many others) were hoping for. Thanks so much for your willingness to jump in like that.
As much as he'd like to make it like we don't fix bugs, our public bug tracker is one click away and shows that out of the 25 open issues and the 493 closed issues, none mentions even remotely a pending crash that he is experiencing. He does have some feature requests and bugs though, with many handled already, so it seems that he does knows where to find it when he needs to.
As for his concerns, yes he did tell me, he sent me 16 mails yesterday, including the one with the trace. Unfortunately, he sent it past midnight, at about the same time he made it public because he just couldn't wait until morning to disclose.
Haven't you been paying attention? You're supposed to write your security-critical code in a language that was released 5 months ago and has no major deployments yet.
The OpenBSD folks have done great work through auditing and writing safer code in C, and I agree that using a language that ameliorated at least some of the common mistakes programmers make would be a net positive.
Related - this blog from @tedunangst is worth reading, regarding the limit to which even a language like Rust can protect against specific exploits (the OpenSSL "Heartbleed" vulnerability, in this case) http://www.tedunangst.com/flak/post/heartbleed-in-rust
1) Rust's use of LLVM means it will not work on some hardware platforms that the OpenBSD folks are committed to run on.
2) The Rust project currently does not offer a long term support release.
These are important to the OpenBSD team. Additionally, I am sure that they value the memory safety that Rust offers.
I wonder if the OpenBSD team would ever design a systems language?
If it works out there, OpenBSD could audit and import things at their leisure, at least in theory.
Here's an example from one of the leading groups in the field:
http://www.cs.cornell.edu/jif/
That was turned into SIF, Swift, Fabric, and I think Civitas voting system. Swift even auto-partitions the system between client and server side while maintaining confidentiality/integrity policies. Fabric kicks butt on all kinds of issues in the hardest space of them all. So, given the above, I'm sure a problem like Heartbleed in a mere SSL protocol would be trivial to prevent at language level if software and compiler developers merely chose to adopt info-flow security at least for important projects.
Note: One can use stuff like that even if not supported in main tool set by creating a mock-up of software in supporting toolset. The abstraction gap presents issues but a problem in one might mean problem in other. Especially for what amounts to covert channel analysis of software.
1. Object X frequently removes limbs from the user.
2. Object Y does the same thing, but more safely.
3. Wait, Object Y causes hangnails if you misuse it hard enough.
4. It's OK to keep using Object X!
Gave details here on C and UNIX history with counterexamples from other systems:
So while I don't like C (or UNIX for that matter) I still do celebrate the fact that these people are actually doing something and even more importantly they are doing what they want despite the naysayers and doing it well.
Some variants of Forth?
OpenSMTPD actually dates back earlier to around 2009, when it was introduced in 4.6.
Not that this is even notable, as virtually all major OpenBSD projects are C-based.
I want to respect OpenBSD. To a great extent, I do. And there are reasons to prefer tested C codebases to untested code bases in new languages, sure. But to the extent I can't give my full, unadulterated respect to OpenBSD, it is because I don't see that they consider everything being in C a problem to be solved.
I mean, I get engineering considerations and that we can't rewrite everything at the drop of a hat and performance matters, etc. But it is my opinion that any engineer that cares about security really ought to see "written in C" as starting out with a huge entry on the debit side, and take appropriate actions. And every year that goes by that's more true.
On the other hand, the issue is a tad more subtle. There seems to be an almost Pavlovian revilement and panic against C code, and this usually comes from people who haven't dealt with the language in a long time or at all in any depth.
For one thing, the blanket assertion that secure C code is a Herculean task, doesn't quite pan out. djb showed one way, based on a few general principles espoused in qmail, that has since successfully been reapplied by people like Wayne Marshall, Laurent Bercot, Bruce Guenter, Paul Jarc, JdeBP and so on. Much of it comes from abandoning the C standard library in favor of more robust alternatives like libdjb variants, clearly architecting process/module boundaries, using simple formats like netstrings and text0 that need not be "parsed" in the conventional sense, etc.
Further along the line, we have the SEI CERT C Coding Standard that comprehensively identifies many pitfalls. [1]
Deeper still, cryptography is one area where abstraction can be an enemy. You actually have to deeply worry about time complexity, branch prediction, precise timing, compiler optimizations and so on [2]. A tool like C becomes desirable here.
Lastly, but crucially - you're on Unix, for better or for worse. POSIX, SUS, all of those standards? They're deeply written with the conventions and API/ABI semantics of C in mind. Such is life. When you primarily write code at that level, the easiest way not to have any impedance mismatch is to just write it in C.
[1] https://www.securecoding.cert.org/confluence/display/c/SEI+C...
For crypto, at this point, C is now too abstracted from the hardware to be useful. You need assembler, and that's getting dangerously abstract too, since it's actually an abstract language over the microcode that actually execute, the entire point of which is generally to provide lovely, wonderful things... that trash timing guarantees that crypto would like.
"I had to write code that allows remote attackers to own the machine because it was the only way to avoid impedence mismatch."
Incidentally, since it hasn't come up, note that I consider "C + useful static analysis tools" to be a different language. "C + useful static analysis tools" can be a sensible choice. I still at least have questions about the cognitive load being loaded on to the developer vs. a language that removes the problems entirely, but it must be acknowledged as having radically different cost/benefit profiles. However, I observe that very few people seem to be using useful static analysis tools beyond -Wall.
Concurrent Pascal was an interesting project for its time, heavily dated by now. The greatest victory for concurrency was having architectures with thread-level parallelism instead of the dominant instruction-level parallelism, and one specifically optimized for CSP in the occam language. This was, of course, the transputer. So much for that.
Erlang/OTP leads in the modern era.
But I don't see any relation to our current topic of discussion.
Think all the race conditions, deadlocks, etc that have hit server apps would've happened on Solo, Ada tasking, or SCOOP? Probably fewer to none. So, it's still relevant and Ada/Eiffel are actually building on it. Especially SCOOP R&D community.
Both OpenSMTPD and qmail seem to rely on concurrency through forkexec, which would entail deferring to the kernel's CPU scheduling and job control.
"Event loops seem to have been the norm for C projects for a while."
I'll take you word for it as I've seen that in the few, top-tier ones I scrutinized. There's a similar trail of research dealing with that alternate model. Have a feeling the meme might apply there, too, but won't accuse them until I see for sure. ;)
To a first approximation, every single one of those wouldn't happen in a memory-safe language.
Of course, plenty of other things will. But it's not like choosing memory-unsafety makes those other things less like (memory unsafety does not, for instance, prevent incorrect ACL specification), so that's not very compelling.
Think of it this way. Suppose C didn't exist, and somebody proposed a new language today that was memory unsafe. Would it have any success, or be laughed out of existence?
In fact, go ahead and suppose C did exist, because it would still not be taken seriously.
If it's so wonderful that C is memory-unsafe, where's all the new languages that are also memory-unsafe?
(Note having specially-called out "unsafe" operators does not make a language "unsafe" the way C is.)
C being a plague still stands.
Compare new to new and tell me: how many web, smtp, imap, dns servers in Python, C# or Rust do we run today? There were some attempts but very few successful ones, despite of all the goodness that comes with better languages, libs and tools. I bet there's a reason behind it, other than just "people didn't try hard enough". There's old stuff out there being used today that we should definitely phase out. But we'll replace system software with new one written in system languages. Most likely C, perhaps Rust. But definitely now with ones written in "safe" languages that should deliver (if they are so awesome) but somehow didn't.
I've already detailed some of what was known about avoiding reliability and security problems before C and UNIX in the link below.
https://news.ycombinator.com/item?id=10273656
I know at least MULTIC's tricks were known at time of UNIX because it's name is a pun on MULTICS. Anyway, the better parts were largely eliminated to make C and UNIX efficient enough to run on a PDP-11. So, it's a fact that good stuff was known, they stripped it out due to constraints, and they never fixed those decisions as constraints lightened. That language, toolset, and coding style were adopted by many developers that continued to do things that way.
Meanwhile, in parallel, other trends began running. The evidence by the 1970's suggested that building secure systems require specs of behavior, a security policy, analysis one possesses the other, careful choice of language, pentesting, minimal TCB's, interface protection, and so on. Starting with MULTICS security evaluation, which talked about "insufficient argument validation" (ex: buffer overflows), that crowd demonstrated in one project and paper after another how to build secure systems. Plus, where most bang for buck was in terms of efforts.
In parallel, people like Margaret Hamilton, Dijkstra, Wirth, Hansen, etc determined how to make software that was correct by construction. Hamilton did that in 60's on Apollo. Dijkstra's method of layering of simple components with careful interfaces made nearly flawless software. Wirth & Jurg did a whole PC, safe (but efficient) language, compiler and OS in 2 years that performed fine. Kept building on that. Hansen and Dijkstra built things like Concurrent Pascal that eliminated concurrency errors on top of Wirth-style work that reduced sequential errors. One group after another built on such approaches with better results than UNIX and C crowds in terms of functionality vs labor available vs defects found.
So, this stuff was well-known at repeated points in the implementation of UNIX, server software, compilers, etc. Better approaches are finally mainstreaming with likes of Go (Wirth-inspired) and Rust (Ada redoux?). Other safe languages were plenty mature before they showed up. Yet, many projects focused on security still use C. So what I'm talking about not only happened then: it's still happening now.
"Compare new to new and tell me: how many web, smtp, imap, dns servers in Python, C# or Rust do we run today?"
Let me just apply your words to another problem. How much enterprise transaction processing uses C and UNIX today vs IBM and COBOL? There were some attempts to switch but very few successful ones despite all the goodness that comes with better languages, libs, and tools. I bet there's a reason behind it. Yep, but it's not that IBM and COBOL were best. I'd venture people not willing to put that work in, fear of breaking things, inertia, not knowing about better approaches, etc. The usual. Currently happening to C, C++, and Java stacks. :)
"But definitely now with ones written in "safe" languages that should deliver (if they are so awesome) but somehow didn't."
They did. Wirth made a whole series of them that delivered in many ways. Borland Delphi, Oberon, Component Pascal, and Modula-3 saw use in industry. GEMSOS used Pascal w/ coding guidelines to hit high security in the 1980's. Ada improved defect count in every empirical study I saw. A tasking profile for Ada and Eiffel both knocked out plenty concurrency errors. MULTICS used PL/0 to avoid issues that plagued C. Academics working on large projects like compilers and sometimes OS's had positive experiences with the LISP's, ML's, and Haskell. And so on.
They proved their worth every time they were used. People's gripes could've been fixed with less effort than what's been put into making C code maintainable, reliable, secure, etc. Instead, they kept trying to build modern stuff on a language & OS stripped of its best features to run on a PDP-11. And stayed on that bandwagon to the end.
(Edit: Corrected by vezzy-fnord, started in 2009.)
The only currently-extant memory-unsafe language.
I... just... argh... when we will finally be rid of this albatross? What hope is there for the world of computer security if we can't even greenfield a solution to what has already proved to be one of the most dangerous security domains there is (thanks, sendmail) in something other than C? Why are the words "buffer overflow" appearing in relation to such young code? What am I doing with my life? Why am I even trying to write secure code in a world where this is even remotely considered an option?