Then again, I'm not sure exactly what a type would be for Unicode. Basically just a flag that the array of bytes has been checked and is valid? It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that.
[0] https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...
In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously.
It's nice in code that maybe isn't very concerned with such things to be able to know that any "string" is actually text we can display, output to a console, something like that. Not for example a TCP/IP packet we haven't decoded yet, or the first 32 bytes of a JPEG image.
Programmers who aren't writing firmware for a vacuum cleaner or washing machine, probably want string literals like "this" in their code, and you'd naturally want those to have a type, for which a string type is the obvious fit. I believe in 2021 this type should obviously be UTF-8. "It's just some bytes" is far from useless, but it isn't much of a string type.
I was sceptical at first about Rust's choice to build in str (immutable UTF-8 string slices), because that seems like a relatively high level concept. But unlike std::string::String, str isn't that tricky after all. See, the only place such things would come from (not having std::string::String to make new ones) is the program source, and our compiler is necessarily already reading the program source, so it follows that the compiler must know how the source is encoded etc and thus it does actually know exactly what those literal strings are. If your literal wasn't Unicode, that wasn't a Rust program and it won't compile.
And also as long as all supported operations on those bytes still result in utf-8 strings (there could be unsafe operations too, like splitting at a random byte offset, but there should be a clearly marked safe set that wont corrupt a valid utf-8 byte sequence).
I have this internally in my lang AST Rust
Utf8
struct City {
name:Utf8
}
Well, as you said "Unicode is quite large and complicated".
And in 2021 it's not just anglosaxon software users anymore (if they ever were), good unicode lib if not type is a must.
You don't need a string type for that, you just need routines that handle UTF-8 strings, like utfcpp (https://github.com/nemtrif/utfcpp).
However, if the philosophy here is to be unopinionated and optimize for flexibility over everything else, I feel I can't really fault them for making that choice.
Like in any language with any lib that is not self-contained (cuts across all aspects of a program and can be used anywhere) and has many different implementations and no sanctioned one.
I was doing some pretty gnarly high-level filesystem work (transforming an unstructured tree of files/folders into a more structured tree following certain naming and categorization conventions -- a lot of parsing/building/concatenating paths). I have been working with higher-level languages for the past decade or so, but have had a wanderlust to get away from garbage collected langauges and go back to my roots a bit, so I've been comparing some of the newer options in the "improved C"/"C++ replacement" space.
It may very well be that I'm not the target demographic for Zig, and that's totally fair, but to me it felt pretty jarring to go back to working with C-style character (byte) arrays and all the (well-known) issues with which they are fraught. I didn't feel like Zig was an improvement over C in the string handling area (as opposed to all other areas), whereas languages like Go and Rust are.
Other than that, I felt Zig was a joy to use.
Rust's OsStr is my favorite approach so far. It stores Linux's raw bytes as-is, and stores Windows's possibly-valid UTF-16 as WTF-8. This makes path management "just work", with the ability to operate normally on invalid UTF-8 or UTF-16 paths, and zero-copy conversion from UTF-8/ASCII strings to OsStr (though converting OsStr into UTF-16 requires parsing). (Qt's QString-based file dialogs on Linux fail to convert invalid UTF-8 paths like those in https://github.com/petrosagg/wtfiles into QString, causing Qt-based apps to open/save the wrong paths.)
However there are difficulties in printing an OsStr. For example, a file dialog that shows filenames as raw bytes can't show non-Latin/Unicode characters in a human-readable form, and a file dialog that shows filenames as Unicode strings can't handle invalid Unicode filenames. GTK3 file dialogs show filenames as Unicode strings, and when encountering files with invalid Unicode names, instead displays "file�name.txt (invalid encoding)".
Worse yet, how should a file dialog allow users to rename files? If it's based around byte arrays, the user can't enter Unicode characters directly, and if it's based around Unicode (or a locale-specific text encoding), it can't display existing files with invalid Unicode/etc. in the name (probably not an issue if it allows the user to rename to a valid name), nor allow users to enter invalid Unicode (which is not an issue IMO).
APFS is utf8, HFS+ is utf16.
> with valid Unicode glyphs
That doesn’t really mean anything.
Apple’s fs do guarantee the paths are correct, as in, valid whatever encoding this has nothing to do with glyphs.
APFS also does not perform any normalisation while HFS+ uses a custom variant if NFD. While HFS+’s normalisation has its issues and critics, APFS’s lack of normalisation is probably worse: https://eclecticlight.co/2017/04/06/apfs-is-currently-unusab...
For a minimal std, maybe Zig can have a standard `str` type which is used for all functions that don't need to grow the string and the `String` type is provided by any 3rd party library which is expected to provide a conversion function from `String` to `str`.
These strings types could or could not have a requirement of being Utf-8. Maybe there is a builtin wrapper to convert them. So you can any dynamic byte array which can be converted to `str` (just a pointer and length, no validation) which can be later converted to `utf8` (str but validated). A `String` which guarantees Utf-8 could directly convert to `utf8` to save validation costs.
The only thing I've found being a pain point is C's biggest pain point: the lack of a proper string type. Even though I've worked for years in the past with C and C++, I still get tripped up, and in Zig I keep on being confused whether I should use raw byte arrays, or sentinel-terminated arrays for certain string manipulations. Maybe I'm not familiar with common idioms in Zig that might help avoid this C-style tracking of string/buffer lengths malarkey that I tend to end up with where a slice is not appropriate.
Maybe a dynamic-length string library is the way forward.
I've always tried as much as possible to treat strings as just opaque data and never look into them, which tends to work well, but in some domains you really need to look at and massage the characters/codepoints/grapheme clusters, and the lack of a first-citizen UTF-8-aware string type is, I think, a bit unfortunate in this day and age. I understand having one of those could make C interop a bit gnarlier (I think Odin's approach of having two separate string types -- one of which is just meant to be used for interop -- should be workable).
Oh, and the ecosystem of libraries is still young. I needed to format timestamps as human readable date/time to a file, and had to step down to C to do something that basic[1], but it'll get there in time, I'm sure.
[1]: "basic" from the perspective of the end-user of the language. Having written a (not even fully featured) date/time library in C++, I consider date and time to be one of the hardest domains I've ever had to work in.