News for hackers. Stuff that matters.
Re:API design note: Beware of adding an "Other" enum value by remram on Sunday March 02, 2025 @03:15PM (#43234554)
Rust has the "non_exhaustive" attribute that lets you declare that an enum might get more fields in the future. In practice that means that when you match on an enum value, you have to add a default case. It's like a "other" field in the enum except you can't reference it directly, you use a default case.

IIRC a secret 'other' field (or '__non_exhaustive' or something) is actually how we did thing before non_exhaustive was introduced.

Re:API design note: Beware of adding an "Other" enum value by kibwen on Sunday March 02, 2025 @04:06PM (#43235096)
Note that the stance of the OP here is broadly in agreement with what Rust does. His main objection is this:

> The word “other” means “not mentioned elsewhere”, so the presence of an Other logically implies that the enumeration is exhaustive.

In Rust, because all enums are exhaustive by default and exhaustive matching is enforced by the compiler, there is no risk of this sort of confusion. And then the fact that his proposed solution is:

> Just document that the enumeration is open-ended

The non_exhaustive attribute is effectively compiler-enforced documentation; users now cannot forget to treat the enum as open-ended.

Of course, adding non_exhaustive to Rust was not without its own detractors; it usage for any given enum fundamentally means shifting power away from library consumers (who lose the ability to guarantee exhaustive matching) and towards library authors (who gain the ability to evolve their API without causing guaranteed compilation errors in all of their users (which some users desire!)). As such, the guidance is that it should be used sparingly, mostly for things like error types. But that's an argument against open-ended enums in general, not against the mechanisms we use to achieve those (which, as you say, was already possible in Rust via hacks).

Re:API design note: Beware of adding an "Other" enum value by tyre on Sunday March 02, 2025 @04:08PM (#43235120)
Maybe there should be a compiler option or function to assert that a match is exhaustive. If the match does not handle a defined case, it blows up.
Re:API design note: Beware of adding an "Other" enum value by aecsocket on Sunday March 02, 2025 @04:14PM (#43235182)
Rust already asserts that a match is exhaustive at compile time - if you don't include a branch for each option, it will fail to compile. This extends to integer range matching and string matching as well.

It's just that with #[non_exhaustive], you must specify a default branch (`_ => { .. }`), even if you've already explicitly matched on all the values. The idea being that you've written code which matches on all the values which exist right now, but the library author is free to add new variants without breaking your code - since it's now your responsibility as a user of the library to handle the default case.

Re:API design note: Beware of adding an "Other" enum value by ffminus on Sunday March 02, 2025 @07:40PM (#43237013)
Library users can force a compile error when new variants get added, using a lint from rustc. It's "allow" by default, so it's opt-in.

https://doc.rust-lang.org/rustc/lints/listing/allowed-by-def...

Re:API design note: Beware of adding an "Other" enum value by WiSaGaN on Sunday March 02, 2025 @10:34PM (#43238040)
Does this require nightly? If so, #[warn(clippy::wildcard_enum_match_arm)] will do the samething but no need for nightly, and from clippy instead of rustc natively.
Re:API design note: Beware of adding an "Other" enum value by codetrotter on Sunday March 02, 2025 @10:56PM (#43238150)
That's pretty neat. I still don't completely understand why #[non_exhaustive] is so desirable in the first place though.

Let's say I am using a crate called zoo-bar. Let's say this crate is not using non-exhaustive.

In my code where I use this crate I do:

  let my_workplace = zoo_bar::ZooBar::new();
  
  let mut animal_pens_iter = my_workplace.hungry_animals.iter();
  
  while let Some(ap) = animal_pens_iter.next() {
      match ap {
          zoo_bar::AnimalPen::Tigers => {
              me.go_feed_tigers(&mut raw_meat_that_tigers_like_stock).await?;
          }
          zoo_bar::AnimalPen::Elephants => {
              me.go_feed_elephants(&mut peanut_stock).await?;
          }
      }
  }
I update or upgrade the zoo-bar dependency and there's a new enum variant of AnimalPens called Monkeys.

Great! I get a compile error and I update my code to feed the monkeys.

  diff --git a/src/main.rs b/src/main.rs
  index 202c10c..425d649 100644
  --- a/src/main.rs
  +++ b/src/main.rs
  @@ -10,5 +10,8 @@
             zoo_bar::AnimalPen::Elephants => {
                 me.go_feed_elephants(&mut peanut_stock).await?;
             }
  +          zoo_bar::AnimalPen::Monkeys => {
  +              me.go_feed_monkeys(&mut banana_stock).await?;
  +          }
         }
     }

Now let's say instead that the AnimalPen enum was marked non-exhaustive.

So I'm forced to have a default match arm. In this alternate universe I start off with:

  let my_workplace = zoo_bar::ZooBar::new();
  
  let mut animal_pens_iter = my_workplace.hungry_animals.iter();
  
  while let Some(ap) = animal_pens.next() {
    match ap {
      AnimalPen::Tigers => {
        me.go_feed_tigers(&mut raw_meat_that_tigers_like_stock).await?;
      }
      AnimalPen::Elephants => {
        me.go_feed_elephants(&mut peanut_stock).await?;
      }
      _ => {
        eprintln!("Whoops! I sure someone notices this default match in the logs and goes and updates the code.");
      }
    }
  }
When the monkeys are added, and I update or upgrade the dependency on zoo-bar, I don't notice the warning in the logs right away after we deploy to prod. Because the logs contain too many things no one can go and read everything.

One week passes and then we have a monkey starving incident at work.

After careful review we realize that it was due to the default match arm and we forgot to update our program.

So we learn from the terrible catastrophe with the monkeys and I update my code using the attributes from your link.

  diff --git a/src/main.rs b/src/main.rs
  index ecc2b89..dbe9490 100644
  --- a/src/main.rs
  +++ b/src/main.rs
  @@ -3,6 +3,8 @@
     let mut animal_pens_iter = my_workplace.hungry_animals.iter();
   
     while let Some(ap) = animal_pens.next() {
  +    #![feature(non_exhaustive_omitted_patterns_lint)]
  +    #[warn(non_exhaustive_omitted_patterns)]
       match ap {
         AnimalPen::Tigers => {
           me.go_feed_tigers(&mut raw_meat_that_tigers_like_stock).await?;
  @@ -10,8 +12,12 @@
         AnimalPen::Elephants => {
           me.go_feed_elephants(&mut peanut_stock).await?;
         }
  +      AnimalPen::Monkeys => {
  +        // Our monkeys died before we started using proper attributes. If they are hungry it means they have turned into zombies :O
  +        me.alert_authorities_about_potential_outbreak_of_zombie_monkeys().await?;
  +      }
         _ => {
  -        eprintln!("Whoops! I sure someone notices this default match in the logs and goes and updates the code.");
  +        unreachable!("We have an attribute that is supposed to tell us if there were any unmatched new variants.");
         }
       }
     }
And next time we update or upgrade the crate version to latest, another new variant exists, but thanks to your tip we get a lint warning and we happily update our code so that we won't have more starving animals.

  diff --git a/src/main.rs b/src/main.rs
  index dbe9490..1e7085a 100644
  --- a/src/main.rs
  +++ b/src/main.rs
  @@ -16,6 +16,9 @@
           // Our monkeys died before we started using proper attributes. If they are hungry it means they have turned into zombies :O
           me.alert_authorities_about_potential_outbreak_of_zombie_monkeys().await?;
         }
  +      AnimalPen::Capybaras => {
  +        me.go_feed_capybaras(&mut whatever_the_heck_capybaras_eat_stock).await?;
  +      }
         _ => {
           unreachable!("We have an attribute that is supposed to tell us if there were any unmatched new variants.");
         }
But what was the advantage of marking the enum as #[non_exhaustive] in the first place?
Re:API design note: Beware of adding an "Other" enum value by ffminus on Monday March 03, 2025 @12:22AM (#43238540)
It lets you have a middle ground, with the decision of when breaking happens left up to library users. Without non_exhaustive, all consumers always get your second scenario. With non_exhaustive, individual zoos get to pick their own policy of when/if animals should starve.

Each option has its place, it depends on context. Does the creator of the type want/need strictness from all their consumers, or can this call be left up to each consumer to make? The lint puts strictness back on the table as an opt-in for individual users.

Re:API design note: Beware of adding an "Other" enum value by saagarjha on Tuesday March 04, 2025 @05:22AM (#43252790)
Swift does this with unknown default.
Re:API design note: Beware of adding an "Other" enum value by kelnos on Monday March 03, 2025 @12:34AM (#43238596)
Consider a bit of a different case. I run a service that exposes an API, and some fields in some response bodies are enums. I've published a Rust client for the API for my customers to do, and (among other things) it has something like this:

    #[derive(serde::Serialize, serde::Deserialize)]
    pub struct SomeEnum {
        AValue,
        BValue,
    }
My customers use that and all is well. But I want to add a new enum value, CValue. I can't require that all my customers update their version of my Rust client before I add it; that would be unreasonable.

So I add it, and what happens? Well, now whenever my customers make that API call, instead of getting some API object back, they get a deserialization error, because that enum's Deserialize impl doesn't know how to handle "CValue". Maybe some customer wasn't even using that field in the returned API object, but now I've broken their code.

Adding #[non_exhaustive] means I at least won't break my customers' code when I add a new enum value.

Re:API design note: Beware of adding an "Other" enum value by sophacles on Monday March 03, 2025 @11:23AM (#43243334)
It's really nice when doing networking protocols and other binary formats. Lots of things are defined as "This byte signifies X : 0 == Undefined, 1 == A, 2 == B, 3 == C, 4-127 == reserved for future use, 128-255 vendor specific options".

This allows you to do something like:

    #[derive(Clone, Copy)]
    #[repr(u8)]
    #[non_exhaustive]
    pub enum Foo {
        A = 1,
        B,
        C,
    }
    
    impl Foo {
        pub fn from_byte(val: u8) -> Self {
            unsafe { std::mem::transmute(val) }
        }
    
        pub fn from_byte_ref(val: &u8) -> &Self {
            unsafe { std::mem::transmute(val) }
        }
    }
    
    #[cfg(test)]
    mod tests {
        use super::*;
    
        #[test]
        fn conversion_copy() {
            let n: u8 = 1;
            let y = Foo::from_byte(n);
            assert!(matches!(y, Foo::A));
    
            let n: u8 = 4;
            let y = Foo::from_byte(n);
            assert!(!matches!(y, Foo::A) && !matches!(y, Foo::B) && !matches!(y, Foo::C));
            let n2 = y as u8;
            assert_eq!(n2, 4);
        }
    
        #[test]
        fn conversion_ref() {
            let n: u8 = 1;
            let y = Foo::from_byte_ref(&n);
            assert!(matches!(*y, Foo::A));
    
            let n: u8 = 4;
            let y = Foo::from_byte_ref(&n);
            assert!(!matches!(*y, Foo::A) && !matches!(*y, Foo::B) && !matches!(*y, Foo::C));
            let n2 = (*y) as u8;
            assert_eq!(n2, 4);
        }
    }
This lets you have a simple fast parsing of types without needing a bunch of logic - particularly in the ref example. Someone else sent you data over the wire and is using a vendor defined value, or a newer version of the protocol that defines Foo::D? No big deal, you can igore it or error, or whatever else is appropriate for your case.

If you want to define Reserved and Vendor as enum attributes, now you have to have logic that runs all the time - and if you want to preserve the original value for error messages, logs, etc - you can't Repr(u8) and take up more memory, have to do copies, etc.

    #[non_exhaustive]
    pub enum Foo {
        Undefined =0,
        A = 1,
        B,
        C,
        Reserved(u8),
        Vendor(u8),
    }
    
    impl Foo {
        pub fn from_byte(val: u8) -> Self {
            match val {
                0 => Foo::Undefined,
                1 => Foo::A,
                2 => Foo::B,
                3 => Foo::C,
                4..=127 => Foo::Reserved(val)
                128.. => Foo::Vendor(val)
            }
        }
    }
You also need logic to convert back to a u8 now too.

It's not strictly necessary, but it certainly makes some things far more ergonomic.

Re:API design note: Beware of adding an "Other" enum value by sophacles on Monday March 03, 2025 @04:56PM (#43247323)
Looking at my code that works on this stuff - the above is just wrong. I was looking at my failed experimental branch not the actual code that does this. The above is a fun way to introduce all sorts of UB.

Apologies for my pre-coffee brainfarts.

Re:API design note: Beware of adding an "Other" enum value by codetrotter on Monday March 03, 2025 @08:59PM (#43249310)
How does the working code look?
Re:API design note: Beware of adding an "Other" enum value by tialaramex on Sunday March 02, 2025 @10:05PM (#43237872)
Importantly #[non_exhaustive] applies to your users but not you. In the defining crate we can write exhaustive matches and those work - the rationale is that we defined this type, so we should know how to do this properly. Our users however must assume they don't know if it has been extended in a newer version.

#[non_exhaustive] is most popular for the variants of an enumeration but is permissible for published structure types (it means we promise these published fields will exist but maybe we will add more and thus change the size of the structure overall) and for the variants of a sum type (it means the inner details of that variant may change, you can pattern match it but we might add more fields and your matches must cope)

Re:API design note: Beware of adding an "Other" enum value by 3836293648 on Monday March 03, 2025 @12:04AM (#43238469)
Wait what. I thought it existed for FFI purposes, regardless of if that's with C or network protocols. The defining crate getting away with it undermines this.
Re:API design note: Beware of adding an "Other" enum value by tialaramex on Monday March 03, 2025 @06:42AM (#43240802)
No. If you mean "I don't know" that's not non_exhaustive that's "I don't know".

For a network protocol or C FFI you probably want a primitive integer type not any of Rust's fancier types such as enum, because while you might believe this byte should have one of six values, 0x01 through 0x06 maybe somebody decided the top bit is a flag now, so 0x83 is "the same" as 0x03 but with a flag set.

Trying to unsafely transmute things from arbitrary blobs of data to a Rust type is likely to end in tears, this attribute does not fix that.

Re:API design note: Beware of adding an "Other" enum value by airstrike on Sunday March 02, 2025 @03:39PM (#43234815)
[dead]
Re:API design note: Beware of adding an "Other" enum value by layer8 on Sunday March 02, 2025 @04:56PM (#43235624)
The better solution is to have two different “default” cases in the language, one that expresses handling “future” values (values that aren’t currently defined), and one that expresses “the rest of the currently defined values”. The “future” case wouldn’t be considered for exhaustiveness checks.
Re:API design note: Beware of adding an "Other" enum value by mayoff on Sunday March 02, 2025 @05:44PM (#43236061)
Swift allows an enum to be marked `@frozen`, which is an API (and ABI) stability guarantee that the enum will never gain more cases. Apple uses this quite sparingly in their APIs.

Swift also has two versions of a `default` case in switch statements, like you described. It has regular `default` and it has `@unknown default`. The `@unknown default` case is specifically for use with non-frozen enums, and gives a warning if you haven't handled all known cases.

So with `@unknown default`, the compiler tells you if you haven't been exhaustive (vs. the current API), but doesn't complain that your `@unknown default` case is unreachable.

Re:API design note: Beware of adding an "Other" enum value by layer8 on Sunday March 02, 2025 @05:58PM (#43236175)
Ah, thanks, I wasn’t aware of these two “default” variants in Swift.
Re:API design note: Beware of adding an "Other" enum value by SkiFire13 on Sunday March 02, 2025 @05:28PM (#43235926)
What would the "future" default case actually do though? When you're in the past there's no value for it, and the moment you get to the future the values will become part of the "present" and will still not fall under the "future" case. You would need some kind of versioning support in the enum itself, but that's a much bigger change.
Re:API design note: Beware of adding an "Other" enum value by layer8 on Sunday March 02, 2025 @05:42PM (#43236040)
“Future” values only become defined (“present” in your sense) at compile-time, but may occur before that at runtime. Note that this mostly presumes a language with separate compilation, or situations like coding against a remote-API spec, where the server may deploy a newer version but your client remains unchanged. Once you compile against the new spec, you’d get errors/warnings about the new, not explicitly handled values, but your existing binary would nevertheless handle those values under the “future” case.

The issue with traditional “default” cases is that they shadow warnings/errors about unhandled cases, but you’d still want to have some form of default case for forward compatibility.

Re:API design note: Beware of adding an "Other" enum value by eru on Sunday March 02, 2025 @07:00PM (#43236690)
> “Future” values only become defined (“present” in your sense) at compile-time, but may occur before that at runtime. Note that this mostly presumes a language with separate compilation, [...]

Separate compilation is a technical implementation detail that shouldn't have an impact on semantics. Especially since LTO (link time optimisation) is becoming more and more common; 'thin' LTO is essentially free in Rust at least in terms of extra build time. LTO blurs the lines between separate compilation units.

On the flip side, Rust can use multiple codegen units even for the same crate, thus introducing separate compilation where a naive approach, like in classic C, would only use a single one.

Re:API design note: Beware of adding an "Other" enum value by layer8 on Sunday March 02, 2025 @07:27PM (#43236905)
Separate compilation is relevant, because it means the version of the interface you compile against may not be the same version you run against. This is fine if the newer version is compatible with the older version. And for the present discussion, we consider an added enum value to not constitute a compatibility break. Nevertheless, it means that the client code can now receive a value that it couldn’t receive before. And it’s useful to be able to define a case distinction for such unknown future values, while at the same time having the compiler check that all currently defined values have been duly considered.

In other words, you want to ensure that you have the most appropriate behavior for whatever values are currently known, and a fallback behavior for the future values that by definition you can’t possibly know at the present time. Of course, this is more or less only practical in languages where the interface version you compile against is only updated deliberately, while the implementation version at runtime can be any newer compatible version.

Re:API design note: Beware of adding an "Other" enum value by sunshowers on Sunday March 02, 2025 @04:29PM (#43235348)
There is currently a missing middle ground in stable Rust, which is to lint on a missing variant rather than fail compilation. There's an unstable option for it, but it would be very useful for non-exhaustive enums where consumers care about matching against every known variant.

You can practically use it today by gating on a nightly-only cfg flag. See https://github.com/guppy-rs/guppy/blob/fa61210b67bea233de52c... and https://github.com/guppy-rs/guppy/blob/fa61210b67bea233de52c...

Re:API design note: Beware of adding an "Other" enum value by eru on Sunday March 02, 2025 @06:57PM (#43236670)
Couldn't clippy do that for you?
Re:API design note: Beware of adding an "Other" enum value by sunshowers on Sunday March 02, 2025 @07:08PM (#43236746)
Not at the moment. The unstable lint is implemented in rustc directly, not in clippy, though I guess it could move to clippy in the future.
Re:API design note: Beware of adding an "Other" enum value by rendaw on Sunday March 02, 2025 @10:00PM (#43237844)
I absolutely _hate_ this. Since you're forced to add a default case, if a new field is added in the future that you need to actively handle it won't turn into a compile error _or_ surface as a runtime error.

I think half of it is developers presuming to know users' needs and making decisions for them (users can make that decision by themselves, using the default case!) but also a logic-defying fear of build breakage, to the point that I've seen developers turn other compile errors into runtime errors in order to avoid "breaking changes".

Re:API design note: Beware of adding an "Other" enum value by bobbylarrybobby on Sunday March 02, 2025 @11:29PM (#43238308)
I agree, this is the one place where upstream crates should be allowed to make breaking changes for downstream users. As a consumer of another crate’s enum, it's easy to enough opt into “never break my code” by just adding default cases, but I'd like to have to opt into that so that I'm notified when new variants are added upstream. Maybe this should even be a Cargo.toml setting — when an upstream crate is marked non-exhaustive, the downstream consumer gets to choose: require me to add default cases (and don't mark them as dead code), or let me exhaustively match anyway, knowing my match statement might break in the future.
Re:API design note: Beware of adding an "Other" enum value by michaeljsmith on Sunday March 02, 2025 @10:04PM (#43237867)
Not sure about Rust, but Typescript allows you to have the default handling but still flag a compile error if a new field is added (the first is useful e.g. if a separate component is updated and starts sending new values).

https://stackoverflow.com/a/39419171/974188

Re:API design note: Beware of adding an "Other" enum value by Spivak on Sunday March 02, 2025 @10:47PM (#43238115)
https://news.ycombinator.com/item?id=43237013

You have to opt into it but it's nice that it's available.

Re:API design note: Beware of adding an "Other" enum value by rendaw on Sunday March 02, 2025 @11:47PM (#43238398)
Oh nice! That seems so backwards, but hey, if it works...
Re:API design note: Beware of adding an "Other" enum value by BlackFly on Monday March 03, 2025 @08:28AM (#43241509)
It is all about what constitutes a major version bump and what constitutes the public api.

If I have a parameter in my public API that has enumerated options, I should be able to add a new option without needing to bump my semver major version number since downstream existing code obviously isn't going to use it yet. If downstream was using my public api's enum for some of their own book keeping and so matched on my enum, I want to reserve the right to to say that that is non-public use of my enum, hence the idea that exhaustiveness in enums is a separate decision on to what is included in a public API or not.

On the other hand, if I introduce a new variant in a return value and existing code will get it and need to actually do something with it, then it should probably be breaking. Errors are somewhat of an exception to this since almost all error enumerations need a general, "unknown error" category anyways and introducing a new variant is generally elevating one case out of that general case. Obviously authors can make mistakes.

The alternative, when you cannot mark non_exhaustive, is to introduce stringly typed catch alls, which is much less desirable for everyone.

Re:API design note: Beware of adding an "Other" enum value by bmoxb on Monday March 03, 2025 @01:45PM (#43245231)
It arguably makes sense in a large monorepo, but otherwise I would agree.
Re:API design note: Beware of adding an "Other" enum value by kpcyrd on Monday March 03, 2025 @06:51AM (#43240856)
It's still a gotcha in Rust, I've seen code like:

  #[non_exhaustive]
  pub enum Protocol {
    Tcp,
    Udp,
    Other(u16),
  }
It allows you to still match on the unrecognized case (like `Protocol::Other(1)`, which is nice), but an additional enum variant may eliminate that case, if our enum gets extended to:

  #[non_exhaustive]
  pub enum Protocol {
    Tcp,
    Udp,
    Icmp,
    Other(u16),
  }
Even though we can add additional variants in a semver-nonbreaking way due to `#[non_exhaustive]`, other people's code may now be broken until they've changed `Protocol::Other(1)` to `Protocol::Icmp`.

Having had this in the back of my head for quite some time, I think instead of an `Other` case there should be two methods, one returns an `Option<Protocol>` and the other one returns the `u16` representation. Unless there's a match on one of your expected cases your default branch would inspect the raw numeric type, which would keep working even if that case is added to the enum.

Re:API design note: Beware of adding an "Other" enum value by remram on Monday March 03, 2025 @02:29PM (#43245699)
You can't use non_exhaustive and Other. If you have an Other then it's exhaustive. This design is wrong.
Re:API design note: Beware of adding an "Other" enum value by ghfhghg on Monday March 03, 2025 @08:37AM (#43241575)
Languages like haxe simply won't compile if you don't cover every enum value in a switch case. Would that not be preferable? I quite like that feature.

F# I believe is similar wrt discriminated unions and pattern matching

Re:API design note: Beware of adding an "Other" enum value by nindalf on Monday March 03, 2025 @08:50AM (#43241682)
I think you misunderstood.

By default Rust expects you to handle every enum variant. Not doing so would be a compile error.

An example - my library exposes enum Colour with 3 variants - Red, Blue Green. In your application code you `match` on all 3. So far so good. But now if I add a 4th colour to my enum, your code will no longer compile because you are no longer handling every enum variant. This is a crappy experience for the user of the library.

Instead, the library writer can make their intent clear - with the #[non_exhaustive] attribute. On such an enum it's not enough to handle the 3 colours of the enum, you must add a wildcard matcher that matches any variants added in future. This gives the library writer flexibility to make changes, while protecting the application developer from breakage.

Re:API design note: Beware of adding an "Other" enum value by ghfhghg on Monday March 03, 2025 @08:43PM (#43249188)
Oh I see. I did indeed misunderstand.

Thanks for taking the time to explain!