Specifically, that the lack of tagged unions (or really, structured return types in general; you can pretend those are a tagged union if you believe hard enough) in C library interop conventions is a terrible foundation on which to build widely-used APIs.

And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.

C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.

"But it has a stable ABI!" So does a land mine. Until you touch it.

you can return a struct just fine... most API's don't though.

Historically it didn't work well between different compilers. Maybe it still doesn't.

Passing and returning structs by value has been part of C since C89. Most compilers pass small structs in individual registers, and nowadays with LTO they'll even elide the copy of bigger ones if they can. What's definitely the case though is that by-value structs weren't a common thing when most of the Unix API was written.

Structure assignment, passing, and returning were added in UNIX V7 C.

My conclusion is that an old tradition of not returning NULL in crypt as a workaround for buggy code which doesn't handle errors was a bad idea.

This is still a consequence of a poor type system.

I don't think so. Even in a language with a good type system like Rust, somebody could have made this API:

    /// Hash the phrase with the given setting.
    fn crypt(phrase: &[u8], setting: &[u8]) -> [u8; CRYPT_OUTPUT_SIZE];

Then, in a later version, `crypt` needs to become fallible, so they'd need to signal errors somehow. But because they really want to preserve backward compatibility, they can't replace the output with a Result type. And so they end up doing the same thing that was done in libcrypt - they output some error value and hackily make sure it's not equal to the `setting` string, under the mistaken assumption that the output of `crypt` would only ever be compared to the `setting` string. And then the same problem happens.

Maybe I'm wrong, but it looks to me like the exact same incentives and ways to screw things up exist in C and in Rust for this particular bug.

You are looking at it from the wrong direction.

The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.

I don't think so. In both Rust and C, the solution is the same - make the function explicitly fallible, instead of trying to preserve backward compatibility. In C, the function would probably just look like this, returning zero on success and negative on failure:

    int crypt2(const char *phrase, const char *setting, char *out);

You can call such a signature "insane" if you want, and that's fine, but it would have averted the bug that is the topic of discussion.

what you are suggesting is a form of type safety. There are still degrees of type safety even in C, and this bug resulted from deciding that backwards compatibility is more important than type safety, which I think we agree on.

And C has effectively deprecated bad APIs over the years. Just not this one.

It's a perfectly sane signature, especially coupled with an API surface that uses the return channel for errors generally.

[delayed]

This isn't good either. Crypt is a pure computation that can't fail for arbitrary reasons. The former implementation could fail if the salt was invalid, which is something you could guarantee. Now it also fails if the password is invalid. That's new behaviour and no matter which way you wrap it up it has the potential to be incompatible. They should have added the maximum length recommendation to the man page, and let DoS attacks continue until the calling software was updated, not the crypt library.

IMO type safety requires both language features and a mindset where you take efforts to keep things type safe. Your example has only the former.

For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.

Yes, even in Lean you can just refuse to prove any properties.

But if you have a stronger and more expressive type system, you are more inclined to use it, instead of looking for silly workaround like in C.

You wouldn't be concerned about breaking backward compatibility with rust though, would you? A recompile would be needed anyway and the change would be detected and enforced. The rationale for the bad API here (two types in one variable) was that users were ignoring NULL - that's the part which wouldn't happen in rust.

Maybe he meant binary compatibility.

They would have to make it crypt2.

OP suggests it was always fallible, they switched to a sentinel to mitigate callers dereferencing null, which they wouldn't have with Option/Result. https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc12

But in your version, you can't preserve backwards compatibility, making an infallible function fallible is a breaking change. For something comparable to libc you might want to (eg Rust std still has mem::uninitialized), but even with security issues, I feel like people would be more inclined to deprecate for a new function (or panic, which does have its own problems).

A good type system also means things are written with better forwards compatibility, and can be safely changed later. The Rust equivalent to `crypt` likely wouldn't exist: a low-level library would just provide hash functions, leaving callers to mitigate DOS etc; a broader library might return some opaque handle with equality/serialization, which you couldn't try to compare to a passwordless user; a high-level library would parse the account store itself. None of these really need to change in incompatible ways, they have all the information to add features and fixes at their own level of abstraction.

The idea of trying to provide a workaround for buggy code calling crypt() is terrifying. For sure, trying to grease the skids for buggy code calling printf(), go for it. But it is hard to imagine cases where slightly-incorrect usage of crypt() is a good idea.

My thoughts exactly.

"Make invalid states unrepresentable" is such a useful adage when designing even basic systems, much less the gates to control your server.

In this particular case, it is type safe. NULL, by the C standard does not refer to an object. It means that every pointer in C is effectively an option type, which mean that a function that returns a pointer can return NULL for "no result", which, in the case of crypt() is perfectly adequate.

It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.

Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.

Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.

You might consider it type safe according to C's rules, but it is not type safe according to contemporary opinion because the success type and error type are the same.

Contemporary opinion among who? Words mean things, and "type safe" simply does not mean "agrees with my preferred style of error reporting"

Papers on type soundness from the last 30 years usually explore the domain further than C semantics.

That's not the same as asserting that soundness requires adherence to the current fashion, sum types over results and errors. TIMTOWTDI.

You don't have to think C is sound or good to distinguish a disfavored API style from mathematical unsoundness. There are infinitely many type safe APIs and languages that are, despite their soundness, ugly and hard to use.

Can you link some?

This is not about soundness. Soundness means that the program behaves in accordance with what its types express. What you're saying is that papers on types show how you can use types to express more constraints.

There can be other valid perspectives and definitions of that word than yours.

I think almost all (tech) arguments stem from simple disagreements and/or different assumptions made on the definition of a (subjective) term.

You may be right in general, but soundness comes from formal logic, where they're pretty fastidious with their definitions. Soundness certainly doesn't mean the expressivity of the types (i.e. what constraints they can represent); it's about the relationship between the runtime behaviour of a program and the types in the code.

Again I think you're trying to push a specific definition as more correct than OP's somehow. Why not just ask them what they meant to understand their intent/context instead of saying they are wrong?

I think I understand their point and even said, "What you're saying is that papers on types show how you can use types to express more constraints," and that's correct. I merely pointed out that that's not soundness, which is something I think they'd like to know. There may be some leeway in the definition of soundness and how it's similar to or different from type safety, but either way, type expressivity is something different altogether (and C certainly has issues with both expressivity and type safety or soundness, but they're different issues). So the point isn't to win - the argument was right and I'm not claiming that C's type system is expressive - but to explain the different terms and their meaning in programming languages and formal semantics.

Opinion among people who know better than to try to explain subjective words with dogma, because that just makes them look silly.

> [...] because the success type and error type are the same.

That's a bit weird? In Rust, if you have an thing that returns Result<String, String> that's still type safe, even though success and error types are the same.

I think I know what you want to express..

You might say it's in-band vs out-of-band error signalling?

> In Rust, if you have an thing that returns Result<String, String> that's still type safe, even though success and error types are the same.

These types are not the same:

    Ok<String>
    Err<String>

Those are not types at all. Ok and Err are type constructors for the Result<A, B> type that take arguments of type A and B respectively.

> type constructors

Nit: the type constructor is `Result`: given two types `A` and `B` (not instances of those types, but actual types) it gives you a new type `Result<A, B>`.

`Ok` and `Err` are called "variants" and can be used to construct a new _instance_ of `Result<A, B>`.

Ah, thanks! I'd remembered the term "data constructor" from Haskell and called it a type constructor instead.

[delayed]

[delayed]

Basically, you need the whole 'success and error types are different' in a language like Python if you make the return type something like "int | string", where 'string' stands for your error message.

Of course that breaks down, when the success type is also an error message, eg when you are looking up something in a catalogue of error messages.

Rust has no trouble with this, for exactly the reasons you line out.

> the compiler can invoke nasal demons if it makes the code go faster

I had never heard that phrase, and was delighted at the answer when I googled it its origin, thank you:

http://www.catb.org/jargon/html/N/nasal-demons.html

> “When the compiler encounters [a given undefined construct] it is legal for it to make demons fly out of your nose” (the implication is that the compiler may choose any arbitrarily bizarre way to interpret the code without violating the ANSI C standard)

However this is limited by the compiler's actual abilities and code. The C standard doesn't forbid nasal demons, but the compiler isn't programmed to invoke nasal demons and isn't running on hardware with that capability, so it won't happen.

I've heard nasal demons invoked as a literal possibility. No they are not a literal possibility.

Maybe your compiler isn't, but GCC is designed to summon every nasal demon it can. It especially loves to elide null pointer checks. There was a PR that almost made it to Linux mainline that did some weird stuff in kernel space where memory mapped to address 0 is valid and contains important stuff. It got a pointer, read data from it, then checked if pointer is null before giving root permissions for some operation. Reading the code, everything seems alright. Even stepping through it with debugger shows that it works. But when compiled with -O2 or higher, GCC removed the null check and now everybody got root permissions unconditionally - because reading from null pointer is UB regardless of circumstances, and GCC treats UB as dead code even if it is reachable.

> GCC is designed to summon every nasal demon it can.

Which is zero, to be clear. There's not even a standard USB device class for a nasal demon releaser

A null check elision is not a demon up my nose - it's a null check elision.

Having made devices for other human holes, you can just use the standardized HID class. It can be adapted for nasal demons incoming and outgoing.

Sure, and char[] is no string because you can't tie anything with it. And don't get me started on the insects that allegedly live inside digital source files - that's an absurd concept, no self respecting computer specialist should be talking like that.

Wait, so if you're running on embedded, you're not allowed to store anything at address 0, requiring you to waste that little bit of RAM?

That's... a bizarre choice, considering how religious C is about not making any assumptions about what a machine can and cannot do, even going to the trouble of working with non-IEEE floats and int types that are not a multiple of 8 bits.

Unfortunately yes. ISO/IEC 9899:1999 footnote 83: "Among the invalid values for dereferencing a pointer by the unary * operator are a null pointer (...)."

No. IIRC, you’re just not allowed to use a constant expression evaluating to 0 as a pointer to construct a non-null pointer to address zero.

There’s no requirement that the in-memory representation of a null pointer be all-zeroes, only that 0 in source code represents whatever it is.

Regardless, I agree that it’s a bizarre choice.

> It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer.

Without the ability to express a non-nullable type, C cannot be unsafe in this way!

Would this mean all pointers in C are type safe?

> It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.

These are not the same, and your second sentence suggests why. Option has but a single error type, but result types and error unions allow arbitrary types of error.

> In this particular case, it is type safe

That depends on how you interpret the pointer type, but in any case there are multiple interpretation and just the fact that these exist will create confusion and type unsafety.

In theory, having newtons and pounds force be represented as `float` is type safe. In practice, it destroyed a multi-million-dollar NASA spacecraft because of human error.

If NASA were to use the equivalent of `pub struct Newtons(f32); pub struct PoundsForce(f32);`, where the types Newton and PoundsForce were not interchangeable, this would never have happened[1].

[1] I'm aware that that story is a bit more complicated than that, that CSV files were involved (which are, again, not type safe), and that, in that particular scenario, more would have to be done to preserve type safety across formats, vendors and codebases, but my point still stands.

I think related to your caveat is:

https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-...

> The newtype is opaque, so the only way to observe it is to convert it back to a [f32].

The implicitness is exactly the problem. The workaround that caused this vulnerability was done because people assumed that crypt() would never return null.

You don't even need a fancy result type. A simple Option<String> would've forced downstream users to consider the failure case, avoiding this situation in the first place.

.unwrap()

Now what?

First you shouldn't have unwraps in prod code, and you can verify that in your CI/build process/whatever. Second, now you have a crash of the SSH server, which is arguably much better than some random logging in as root.

This isn't about "you", it's about whether any consumer of your code is using unwrap on the result, which currently is perfectly safe to do. The equivalent in C is returning NULL, which would also have caused a crash. They decided not crashing was preferable, which is how we ended up here.

[delayed]

I don't really care about the form of the argument when the concrete results are that (i) “no unwrap in prod” is trivial to ensure through the toolchain, (ii) even though, its consequences are “damn it crashed” instead of “welp I guess this guy is root now”.

[delayed]

> And that analogous situation can happen in the safer language where crypt returns an Optional(String) which some bad programs assume is always present.

Once again, no, because these unknown programs would just crash on the `.unwrap()`; not get in a silently corrupted state. Same causes, different effects.

[delayed]

Absolutely not. Unwrap is a language rule, so everyone knows unwrap is not a safe thing to do and writing it even once is an explicit dismissal (and you should document it). On the other side, knowing that crypt does not necessarily return a hash is easily overlooked and forgotten, there is no code smell that can hint you that you are doing things wrong.

[delayed]

I think the argument would be that in C people will look at a char* and think "string" rather than "string or null". Thus consumers of the library not checking the result and the maintainer trying to be helpful by putting a "safe" value in there for the error case.

Whereas if the library returns something that is unambiguously an option type, consumers unambiguously call unwrap at their own risk.

Of course you do have a point (assuming I'm picking up what you're suggesting correctly) that all pointers in C are option types. But I think there's a cognitive ergonomics aspect to this where an option type more clearly relays the library authors' intent.

Now you have an easy way to discover the bug

[delayed]

Exactly, whoever program in C knows that every function that returns a pointer may return NULL (even if not documented) and thus every pointer must be checked.

However a better API (to me) that I would have written was:

    crypt_error_t crypt(...args, char *result, size_t resultLength) 

That way btw the function doesn't need to allocate memory inside, which is (being a mostly embedded programmer) something I don't like at all. Plus returning an error enum makes it clear instead of having to look at errno.

But POSIX API is not something well designed, it dates back to the '80 so... probably today would be written completely differently

[delayed]

That and in-band signalling is error-prone.

Another example that C programmers love: using 0 to mean infinity. Eg when you have a timeout setting, and 10 means ten seconds, but 0 means infinity seconds instead of zero seconds.

That’s just a very bad API. You could do the same in Rust? There are nice C APIs that would use a nice struct that describes the timeout you want… eg GCD.

I'm not sure a struct is the right thing here in general. You often want an algebraic datatype.

What Rust calls an enum, but an enum with payloads. In C terms it would be a discriminated union.

In general it's a problem of sentinel values. Some languages let you abstract this - I think in Rust and C++ you can define an "intrusive optional" which is like optional but uses an invalid value pattern to store the not-present state. In memory it's the same as a sentinel value but the compiler forces you to unwrap it like an optional.

I'm not sure how well the 'forcing' works for Rust and C++. You can definitely do this kind of thing in dependently typed languages.

The type is akin to optional<int>. You can't add 5 to it, you have to dereference it first which crashes if it was null (0 or another chosen value).

Well, in Rust you would use pattern matching, which can't even crash.

In Haskell, the closest thing to adding 5 to an optional value would be something like:

    fmap (5+) my_optional_value

This one leaves 'Nothing' alone, but adds 5 in case the value is present.

Amusingly, Win API of all places generally evades this pattern: 0 timeout means "no waiting", and to wait indefinitely, you use timeout of INFINITE which is actually UINT_MAX or, equivalently, -1 (in Win API, timeouts are unsigned, not time_t).