Zig, Rust, and Other Languages

Re:Zig, Rust, and Other Languages by _nalply on Saturday March 16, 2024 @12:38PM (#39727329)

You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would contend it's undefined behavior. At least I am convinced that you should never have invalid UTF-8 in `&str` or `String`. If you need that, there's the [bstr][0] crate for byte strings which may or may not be valid UTF-8. Essentially these are just byte slices `&[u8]` and the owned version byte vectors `Vec<u8>` and traits and methods around them. Very useful for text data which is not reliably UTF-8.

[0]: https://crates.io/crates/bstr

The idea of using unsafety here is to have a way to tell the Rust compiler: "Trust me, I promise that I only fill the strings with valid UTF-8 data. You don't need to insert code to check UTF-8 validity." And the Rust compiler answers: "Okay. I accept that but if something goes wrong I can't catch that."

This allows avoiding repeated validations of a byte slice for UTF-8 validity, for example.

[ Parent | on HN ]

Re:Zig, Rust, and Other Languages by steveklabnik on Saturday March 16, 2024 @12:49PM (#39727431)

It changed, it used to be immediately considered UB, but isn’t any longer, as your grandparent says, that’s what is documented now. It still leads to UB, but isn’t inherently UB on its own. There’s a term for this but I’m forgetting at the moment and search is failing me.

I agree that bstr is what you want unless you specifically understand this detail and know why it is the way it is.

[ Parent | on HN ]

Re:Zig, Rust, and Other Languages by afdbcreid on Saturday March 16, 2024 @07:17PM (#39730315)

The term is "library UB" versus "language UB". To be clear, you should still avoid it most of the times.

[ Parent | on HN ]

Re:Zig, Rust, and Other Languages by steveklabnik on Saturday March 16, 2024 @11:06PM (#39731572)

Ah yes, thank you.

[ Parent | on HN ]

Re:Zig, Rust, and Other Languages by _nalply on Sunday March 17, 2024 @03:11AM (#39732525)

Would you say library undefined behavior is just a violation of a library invariant? Like `&str` and `String` containing valid UTF-8?

I admit that a library invariant is allowed to be violated temporarily under some strict preconditions. One example are bank account money transfers, where the credit and debit don't happen exactly simultaneously. Something similar could happen in a library.

But for language UB this doesn't work. Why? Is it because of compile-time optimizations possibly deleting code restoring the invariant?

[ Parent | on HN ]

Re:Zig, Rust, and Other Languages by josephg on Monday March 18, 2024 @04:45AM (#39741597)

> You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8.

Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.

[ Parent | on HN ]