> This should be solved in a string processing library
Then a type String also should be defined in a string processing library. Either String follows assumptions of a string processing library, or it doesn't. If it doesn't it makes the task of writing a good processing library much more difficult, and you'll get errors thrown from inside of a string processing library, and you'll need to deal with them.
UTF-8 was specifically designed such that most code can deal with it byte-by-byte without any decoding step. I've written many parsers for example, they all just read byte-by-byte and special things happen at ASCII characters (such as ';' or '\n'), and it works trivially with UTF-8 inputs, I don't have the care at all.
I've also written a text editor with a complicated text rope data structure in it. Do you think I should have made a different text rope for each different text encoding the editor should deal with?
No - what my editor does in UTF-8 mode for example, at the visual and editing layer it pulls out data byte-by-byte from the text rope and interprets it as UTF-8. If there is invalid UTF-8 it has to deal with it. But hey, that is the reality of files on a file system - they can contain encoding errors, right? Deal with them! (if you can't simply ignore them).
> Or when trying to get a slice with chars from 5 to 12.
There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called?
If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type.
The reality is that data is stored in memory in byte sequences, and that's the representation that a programming language should expose. Everything else is code / libraries.
No, I'd think that you should make a text rope for UTF-8, and then change encoding of text on the boundary. Or, it might be not an UTF8 but some other representation of Unicode, it depends. I see no reason to create a structure for an effective manipulation of strings without choosing a representation of a character at the compile time, because otherwise it would be slower than it might be. The more assumptions about your data you've made at the compile time, the less runtime conditioning you'd need, the faster your code would be.
Believe me, I had dealt with different encodings all the time. I'm Russian, and we had three widely used unibyte encodings for a cyrillic, plus different encodings for Unicode. So one had to deal with all of them all the time. The easiest way is to deal internally with Unicode only and to change encoding on the boundaries where your program communicates with an external world. There (on the boundaries) you can deal with errors, like character which cannot be represented in an output encoding (or cannot be represented in an internal one, but if you use Unicode it wouldn't be a problem). You can treat user input as an input in an external encoding and throw errors if she inputs something that cannot be encoded in an output encoding. It works all the time, while making your program able to deal with different internal encodings is a PITA, with errors thrown from the most unexpected places, with a spaghetti code trying to deliver errors to places where these errors could be sensibly dealt with.
> There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called?
You can use all of them, just pick distinctive names for them, like "char", "glyph", ... and any other you like. But when you did it, you'd want to know where are the boundaries of these things. You'd want to make slices of sequences of these things. If you cannot rely on a validness of underlying UTF8 then you'll be in trouble.
> If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type.
When I need to work with bytes, I use an array of bytes. Not a string, but an array of bytes. It was hard to grasp after years of experience with a C, but I did managed it at some point. Char is not a byte. Byte is not a char. Array is not a string, string is not an array. When I need an array to deal with bytes, I use an array. When I need a string to deal with characters, I use a string. It is a non-trivial idea for a C-programmer, because all his experience tells him that character and byte is the same thing. So if character is not a byte, then (he reasons) character doesn't exist.
If characters as codepoints is a too low abstraction for my task, I can create atop of it another abstraction dealing with glyphs, words, tokens, sentences or something. But the abstraction of codepoints must be a library feature, or I'd be forced to create it myself, to validate UTF8 all over the place, and so on. And if that so, then what the point to have an abstraction of string?
If characters as codepoints is a too high abstraction for my task, I can go lower and use an array of bytes.
It is really an easy idea, just C as a language tends to confuse people minds by teaching them that char==int8_t. At least my mind was confused and I managed to untangle that mess completely only around my 30th birthday. And several years later I've found that Rust's std is totally differentiate chars/bytes, strings/arrays as I do. I had fallen in love with Rust immediately.
But that's what I made. UTF-8 is encoded as bytes. I can store the bytes in the rope just fine.
The rope has a very simple API, basically read() and write() functions, just like a standard FILE I/O API. Do you want to pick on file system developers that they should add APIs for write_UTF8(), write_LATIN1(), write_KOI8(), write_BYTES(), etc.? And then go to network API designers to do the same for the socket I/O functions? And so on? Of course you don't do that, that would be very bad factoring.
And it's just the same for a rope API.
> The more assumptions about your data you've made at the compile time, the less runtime conditioning you'd need, the faster your code would be.
This is true in general, but the rope is just a storage. No processing happens there. The rope couldn't care less what things you store there. There is no point of having multiple identical read/write implementations.
But if you insist, I recommend to ask for an UTF-8 optimized HDD at your local computer shop :-)
> making your program able to deal with different internal encodings is a PITA
If your program has to deal with multiple external encodings, either you can convert at the boundaries to a canonical internal encoding, or you can't in which case it probably becomes a little more work since you have to convert at different places.
This has nothing to do with what I said, though.
In this paragraph you seem to be confusing the C's "char" with the much more fuzzy idea of "Character" which has like 13 valid definitions.
C's "char" is abstractly defined as the smallest addressable unit of memory available on the machine (required to have at least 8 bits), and historically there have existed 8-bit, 9-bit, 16-bit, or even 36-bit chars. In today's practice it is universally taken synonymous for (8-bit) bytes since all hardware is 8-bit by now. Some people like to be pedantic about the distinction between byte and char, but I most often do not, especially since char is the generally interoperable type in C (with respect to type punning etc.), while uint8_t to my knowledge is not.
"Character" is sometimes understood as "Unicode codepoint" (typically represented as a 32-bit entity, or even as a UTF-8 encoded slice of bytes) or in some cases understood as "Unicode glyph" (probably represented as a slice of codepoints), sometimes understood as even other things.
This stuff is specifically some of the big design wins of these systems. I recommend you to peek over at the Win32 API for example, with its myriads of *A and *W functions, required compiler settings and macro magic to switch between those (even though of course they can't really paper over the difference), then best practice recommendations that have changed multiple times in history, then strange bugs that appear when some invalid data has sneaked into a system that was assumed "pristine"...
Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.
And a difference between "UTF-8" and arbitrary bytes is insofar sensible in that you can perform text operations on the former, but not on the latter. Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
See here for example: https://docs.microsoft.com/en-us/windows/apps/design/globali...
If you browse around various documentation, you'll see contradicting statements which variants are recommended, which I take as another sign that the idea of making a distinction on the type level is simply a bad idea.
As to the macros, yes, if you give the A or W explicitly, you won't need the macro setting that translates the unspecific names to the A or W variants. And as said, the macros aren't a good idea anyway, as it's still extremely hard to properly abstract the distinction (the types are different sizes!!), so code is generally tied to a specific choice either way.
> Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
It can be stored as a byte stream. Whenever you work with the data, you might need to operate on transformed representations - 32-bit codepoints, or larger combinations, or even more complicated stuff like words, sentences, paragraphs, tags, whatever it needs to do the task as done. This is programming, you transform data to achieve things.
What I say is that it's stupid to make a distinction on the type level between things that are entirely the same thing in memory, and that are going to be used for the same things. It's stupid because it unnecessarily create incompatibilities between data and introduces unnecessary "conversions" / copies and requires more code.
Secondly: if there are worse places than Unix, it doesn't mean that Unix is good.
I mean, I love Unix, but Unix sometimes a nightmare to deal with. And kernel believing that "string" means "a byte array with a zero-terminator" one of the worst things of Unix. If one needs an array of variable size, it might do either:
struct unsized_array_t {
size_t size;
uint8_t bytes[]; // being really a bytes[size]
}
struct array_pointer_t {
size_t size;
uint8_t *ptr;
}
The first one is tricker, you'd need to pass around pointers to it, and slices of it would be a problem. But at least you wouldn't need to scan memory to learn the size of it.
Unix strings are not strings but zero-terminated arrays of non-null bytes. It would be obvious if you try another way to deal with a variable size. The Unix-Fathers had an idea how to deal with string, but their idea was proven bad. A lot of functions they invented to deal with their "strings" are now deprecated or even forbidden. Like `gets` for example. I'd hate to use strcpy, I'd better use strncpy, and other functions with `n` inside. But if you started to juggle not just with pointers to strings but with those n's also, you would think of representing strings as structs with an embedded n. And -- viola -- all the Unix's string crap goes through a window. You end with variable sized arrays, and you doesn't need strings anymore. Without losing anything useful.
> Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.
I like this one. Maybe you are right after all.
C doesn't prevent you from creating such a string type, or just passing a separate length as argument to functions. In fact it's the right thing to do (pointer + length) in many cases.
All C does is offer you string literals that are zero-terminated, which is typically practical. Often those are all you need, i.e. to store a plain-text identifier that doesn't allow for NUL characters anyway - so NUL can be used as a sentinel for efficient storage.
Then there is the C standard library, which is a bag of bad practice. Just ignore 90% of the stuff in there. stuff like strtok() etc. is bad. Mostly you want to use memcpy(), memcmp(), strcmp(), maybe strcpy()/strncpy() etc. Then there's stuff like malloc()/free() and stdio and especially the formatting functions that you can use to get something up and running before possibly replacing them with something better. Well, that's about it:-)
See here, it might be that they recognized that choices made in the 90's were wrong: https://docs.microsoft.com/en-us/windows/apps/design/globali...
"Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes."
> I like this one. Maybe you are right after all.
Thanks :-)
> Hopefully just bytes, which trivially allows storage of UTF-8.
Yes, it trivially allows, but then you'd need to deal with utf8-errors all over the program at runtime. Any method for string would need to be written such a way, that deals with invalid utf8 byte sequences.
Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. It gives a lot of edge cases which a programmer must bear in his/her mind constantly, because these edge cases would choose the least expected moment to jump on you. And then you'd need, for example, to invent a way to output an arbitrary byte sequence into a place where only UTF8 is allowed. I, personally, hate it. Like arbitrary byte sequences as a file names, even when I know that no one uses non-utf8 file names, I need to write programs working with file names in such a way, that allows arbitrary byte sequences, and to devise some syntax to output arbitrary byte-sequence into a terminal. Or into a web-page.
I see no good reasons to replace strings with arbitrary byte-sequences. If you need arbitrary byte-sequences then you have another abstraction for your task: an array. Much more powerful, because it can be array of bytes, of uint16_t, or of int64_t, or of your own struct. Why to spoil an abstraction of string with an ability to deal with arbitrary byte sequences?