News for hackers. Stuff that matters.
Re:Maintain It with Zig by losvedir on Wednesday September 08, 2021 @02:53PM (#28461516)
Note that while strings are generally opaque bytes in Zig, there are standard library functions[0] to work with them as Unicode codepoints and such. But Unicode is quite large and complicated, and things like grapheme clusters aren't in the standard library (yet?). I also believe that that module is planned to be rewritten. So it's more the case that Zig plans to someday support Unicode at the standard library level, but not as a specific type.

Then again, I'm not sure exactly what a type would be for Unicode. Basically just a flag that the array of bytes has been checked and is valid? It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that.

[0] https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...

Re:Maintain It with Zig by tialaramex on Wednesday September 08, 2021 @04:45PM (#28462817)
> Basically just a flag that the array of bytes has been checked and is valid?

In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously.

It's nice in code that maybe isn't very concerned with such things to be able to know that any "string" is actually text we can display, output to a console, something like that. Not for example a TCP/IP packet we haven't decoded yet, or the first 32 bytes of a JPEG image.

Programmers who aren't writing firmware for a vacuum cleaner or washing machine, probably want string literals like "this" in their code, and you'd naturally want those to have a type, for which a string type is the obvious fit. I believe in 2021 this type should obviously be UTF-8. "It's just some bytes" is far from useless, but it isn't much of a string type.

I was sceptical at first about Rust's choice to build in str (immutable UTF-8 string slices), because that seems like a relatively high level concept. But unlike std::string::String, str isn't that tricky after all. See, the only place such things would come from (not having std::string::String to make new ones) is the program source, and our compiler is necessarily already reading the program source, so it follows that the compiler must know how the source is encoded etc and thus it does actually know exactly what those literal strings are. If your literal wasn't Unicode, that wasn't a Rust program and it won't compile.

Re:Maintain It with Zig by coldtea on Thursday September 09, 2021 @12:33AM (#28466221)
>In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously.

And also as long as all supported operations on those bytes still result in utf-8 strings (there could be unsafe operations too, like splitting at a random byte offset, but there should be a clearly marked safe set that wont corrupt a valid utf-8 byte sequence).

Re:Maintain It with Zig by mamcx on Wednesday September 08, 2021 @05:48PM (#28463476)
> Then again, I'm not sure exactly what a type would be for Unicode.

I have this internally in my lang AST Rust

    Utf8
I think that is the type:

    struct City {
        name:Utf8
    }
For a scripting lang like mine I surface it as `Str` but in a system language I think is better to say `Utf8`
Re:Maintain It with Zig by moogly on Wednesday September 08, 2021 @03:06PM (#28461649)
Looks like this is an attempt: https://github.com/jecolon/zigstr
Re:Maintain It with Zig by coldtea on Thursday September 09, 2021 @12:31AM (#28466211)
>*It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that.

Well, as you said "Unicode is quite large and complicated".

And in 2021 it's not just anglosaxon software users anymore (if they ever were), good unicode lib if not type is a must.