> The C and Rust union types are extremely sharp blades

Sure, but the comparable Rust feature is enum, not union.

Unions can be used as a somewhat safer (not safe by any means but safer), more flexible, and less error-prone form of transmute. Notably you can use unions to transmute between a large type and a smaller type.

That is essentially the motivation, primarily in the context of FFI where matching C's union behaviour using transmute is tricky and error-prone.

There are rare cases where all attributes of the C union are valid at the same time. Say you have a 32-bit RGBA color value and you want to access the individual 8 bit values. You can make a union of an 32 bit int and a struct that contains 4x 8 bit integers.

Also you can manually tag them and get s.th. more like other high level languages. It will just look ugly.

Yes. I once wanted C unions limited to fully mapped type conversions, where any bit pattern in either type is a valid bit pattern in the other. Then you can map two "char" to "int". Even "float". But pointer types must match exactly.

If you want disjoint types, something like Pascal's discriminated variants or Rust's enums is the way to go. It's embarrassing that C never had this.

Many bad design decision in C come from the fact that originally, separate compilation was really dumb, so the compiler would fit in small machines.

FYI small string optimizations are generally implemented using unions.

In Rust? The two I'm big fans of, CompactString and ColdString do not use unions although historically CompactString did so and it still has a dependency on smallvec's union feature

ColdString is easier to explain, the whole trick here is the "Maybe this isn't a pointer?" trick, ColdString might be a single raw pointer onto your heap with the rest of the data structure at the far end of the pointer, this case is expensive because nothing about the text lives inline, but... the other case is that your entire text was hidden in the pointer, on modern hardware that's 8 bytes of text, at no overhead, awesome.

CompactString is more like a drop-in replacement, it's much bigger, the same size as String, so 24 bytes on modern hardware, but that's all SSO, so text like "This will all fit nicely" fits inline, yet the out-of-line case has the usual affordances such as capacity and length in the data structure. This isn't doing the "Maybe this isn't a pointer?" trick but is instead relying on knowing that the last byte of a UTF-8 string can't have certain values by definition.

I realise that I don't do the best job of explaining ColdString here. After all most 8 byte strings of UTF-8 text could equally be a pointer so, why can this work?

All ColdStrings which look like 8 bytes of UTF-8 text really are 8 bytes of UTF-8 text, just the type label on those 8 bytes isn't "[u8; 8]" an array of 8 bytes but instead "mut *u8" a raw pointer. "Validate" for example is 8 bytes of ASCII, thus UTF-8, and Rust is OK with us just saying we want a pointer on a 64-bit machine with those bytes. It's not a valid pointer, but it is a pointer and Rust is OK with that, we just need to be careful never to [unsafely] dereference the pointer because it's invalid

OK, so there are two cases left: First, what if there are fewer bytes of text? Zero even?

Since there are fewer than 8 bytes of text we can use the whole first byte to signal how many of the remainder are text, we use the UTF-8 over-long prefix indicator in which the top five bits of the byte are all set, bytes 0xF8 through 0xFF for this, there are eight of these bytes corresponding to our 8 lengths 0 through 7 inclusive. Because it's over-long this indicator isn't itself a valid UTF-8 prefix. Again we can pretend this is a pointer while knowing it's invalid.

Lastly, the seemingly trickiest problem, what if the string didn't fit inline? We use a heap allocation to store the text prefixed by a variable size integer length and we insist this allocation is aligned to 4 bytes. This means a valid pointer to our allocation has zeroes for the bottom two bits, then we rotate that pointer so those bottom two bits are at the top of the first byte position (depending on machine word layout) and we set the top bit. This is now always invalid UTF-8 because it has the continuation marker - the top bit is set but the next is not, which cannot happen in the first byte of any UTF-8 text, and so our code can detect this and reverse the transformation to get back a valid pointer using the strict provenance APIs if this marker is present.

This type is tomtomwombat's idea, credit to them:

https://github.com/tomtomwombat/cold-string*

My bad I thought it still used `union`s. I guess I'm not very up to date

In Rust's case, union types should only be used for FFI with a C ABI.

As for C, it is a sharp blade on its own.

I do not agree. MaybeUninit<T> is without any doubt more valuable than the C FFI use

I can't even think of any prominent C FFI problems where I'd reach for the union's C representation. Too many languages can't handle that so it seems less useful at an FFI edge.

OS APIs for one, at least there are some Win32 calls that take unions if I remember correctly.

One of the reasons .NET had Managed C++, replaced by C++/CLI (nowadays C++20 compliant, minus modules), is exactly that P/Invoke (and RCW/CCW) cannot represent everything.

Which they don't want to expose on .NET type system directly.