"WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair. It represents, in a way compatible with UTF-8, text from systems such as JavaScript and Windows that use UTF-16 internally but don’t enforce the well-formedness invariant that surrogates must be paired."
Though note that it's only suitable for internal usage (as in, you wouldn't expose an API that uses WTF-8), so it's not something that you need to actually think about when learning Rust. :P
1. a plain "buffer of unparsed octets" type (like Erlang's "binary" type), to hold onto your weird invalid UTF-16 (or whatever other weird invalid stuff you like);
and 2. a strict, validated-on-construction "sequence of codepoints" type.
Going from #1 to #2 wouldn't be a direct cast; it'd instead be a stream-parser function that could be hooked with a callback, which would be handed parsed-and-valid, and invalid-and-left-unparsed, "segments":
maybe_parsed_segment() = valid_string_of_codepoints() | binary()
...and would then respond with a valid segment, which could just be "" to discard the input segment.The default callback would presumably respond to a binary() of size N by emitting a "�" string of length N. That's pretty much what we have today, but both sturdier and more extensible.
Implemented: https://github.com/rust-lang/rust/blob/master/src/libstd/sys...