Now just UTF-16 and non '\n' newline types remaining to go

UTF-16 will be quite the mountain as Windows APIs and web specifications/engines default to it for historical reasons.

It's not just Windows and JavaScript. On Apple platforms, NSString is UTF-16. On Linux, Qt uses UTF-16 strings. Looking at languages, we have Java (which is where JS got this bug from) and C# both enshrining it in their respective language specs.

So it's far more pervasive than people think, and will likely be in the picture for decades to come.

ICU (International Components for Unicode, the library published by the Unicode folks) itself uses UTF-16 internally, and most of these things are built on ICU. I agree strongly with your conclusion--UTF-16 isn't going anywhere. I don't think the ICU people are even talking about changing the internals to UTF-8.

Qt could really change if they wanted to and really should have by now - it's not like they keep long-term backwards compatibility anyway unlike the others that you mentioned.

Of course they chose to integrate JavaScript so that's less likely now.

Of the two UTF-16 is much less of a problem, it's trivially[1] and losslessly convertible.

[1] Ok I admit, not trivially when it comes to unpaired surrogates, BOMs, endian detection, and probably a dozen other edge and corner cases I don't even know about. But you can offload the work to pretty well-understood and trouble-free library calls.

It causes such issues as opening of files by filename not being cross platform with standard libc functions since in Windows w-string versions are required since UTF-16 chars can have 0-bytes that aren't zero terminators

It makes sense because UTF-16 is a sequence of uint16_t values, so both bytes must be zero, not just one.

Yes, it makes sense, but it still resulted in a lot of work.

Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose. Even some multi-byte encodings would work if they chose to avoid using 0-value bytes for this reason.

UTF-16LE/BE (and UTF-32 for that matter) chose not to allow for this, and the result is that if you want UTF-16 support in your existing C-string-based syscalls you need to make a second copy of every syscall which supports strings in your UTF-16 type of choice.

> Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose

That's completely wrong. If a syscall (or a function) expects text in encoding A, you should not be sending it in encoding B because it would be interpreted incorrectly, or even worse, this would become a vulnerability.

For every function, encoding must be specified as are specified the types of arguments, constraints and ownership rules. Sadly many open source libraries do not do it. How are you supposed to call a function when you don't know the expected encoding?

Also, it is better to send a pointer and a length of the string rather than potentially infinitely search for a zero byte.

> and the result is that if you want UTF-16 support in your existing C-string-based syscalls

There is no need to support multiple encodings, it only makes things complicated. The simplest solution would be to use UTF-8 for all kernel facilities as a standard.

For example, it would be better if open() syscall required valid UTF-8 string for a file name. This would leave no possibility for displaying file names as question marks.

Why should the OS mess with application data? I think syscalls should treat text as the blob it is and not care about the encoding at all.

File name is a string, not a blob.

Yes and my argument is that the OS should treat strings as a blob and not care about the encoding. How can it know what shiny new encoding the program uses? Encoding is a concern of the program, the OS should just leave it alone and not try to decode it.

The OS treats strings as a blob, yes, but typically specifies that they're a blob of nul-terminated data.

Unfortunately some text encodings (UTF-16 among them) use nuls for codepoints other than U+00. In fact UTF-16 will use nuls for every character before U+100, in other words all of ASCII and Latin-1. Therefore you can't just support _all_ text encodings for filenames on these OSes, unless the OS provides a second syscall for it (this is what Windows did since they wanted to use UTF-16LE across the board).

I've only mentioned syscalls in this, in truth it extends all through the C stdlib which everything ends up using in some way as well.

You're right I missed that. Sounds like blob size should be communicated out of band.

You should not be passing file names in different encodings because other apps won't be able to display them properly. There should be one standard encoding for file names. It would also help with things like looking up a name ignoring case and extra spaces.

I mean, I agree there _should_ be one standard encoding, but the Unix API (to pick the example I'm closest to) predates these nuances. All it says is that filenames are a string [of bytes] and can't contain the bytes '/' or '\0'.

It is good for an implementation to enforce this at some level, sure. MacOS has proved features like case insensitivity and unicode normalization can be integrated with Unix filename APIs.

File name is not a blob because it is entered by the user as a text string and displayed for the user as a text string and not as a bunch of hex digits. Also, it cannot contain some characters (like slash or null) so it's not a blob anyway.

And you should be using one specified encoding for file names if you want them to be displayed correctly in all applications. It would be inconvenient if different applications stored file names in different encodings.

For the same reason, encoding should be specified in libraries documentation for all functions accepting or returning strings.

UTF-16 arguably is Unicode 2.0+. It's how the code point address space is defined. Code points are either 1 or 2 16-bit code units. Easy. Compare w/ UTF-8 where a code point may be 1, 2, 3, or 4 8-bit code units.

UTF-16 is annoying, but it's far from the biggest design failure in Unicode.

UTF-16 is the worst of all worlds. Either use UTF32 where code-points are fixed, or if you care about space efficiency use UTF8

UTF-32 is arguably even more worst of all worlds. You don't get fixed-size units in any meaningful way. Yes you have fixed sized code points, but those aren't the "units" you care about; you still have variable size grapheme clusters, so you still can't do things like reversing a string or splitting a string at an arbitrary index or anything else like that. Yet it consumes twice the space of UTF-16 for almost everything, and four times the space of UTF-8 for many things.

UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything.

As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.

In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format.

There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.

Yeah, you need some kind of sequence of Unicode scalars. But there's no reason for that sequence to be "a contiguous chunk of memory filled with 32-bit ints" (aka a UTF-32 string); it can just as well be an iterator which operates on an in-memory UTF-8 string and produces code points.

> It's how the code point address space is defined.

Not really. Unicode is still fundamentally based off of the codepoints, which go from 0 to 2^16 + 2^20, and all of the algorithms of Unicode properties operate on these codepoints. It's just that Unicode has left open a gap of codepoints so that the upper 2^20 codepoints can be encoded in UTF-16 without risk of confusion of other UCS-2 text.

You forgot `- 2^11` for the surrogate pairs. Gee, why isn't Unicode 2^21 code points? To understand the Unicode code point space you must understand UTF-16. The code space is defined by how UTF-16 works. That was my initial point.

If you're going to count the surrogate pairs as not-a-Unicode-codepoint, you should also count the other noncharacters: the last two codepoints on each of the 17 planes and the range U+FDD0-U+FDEF.

The expansion of Unicode beyond the BMP was designed to facilitate an upgrade compatibility path from UCS-2 systems, but it is extremely incorrect to somehow equate Unicode with UTF-16.

FWIW there is an official term for "code points excluding surrogates", it is "Unicode scalar value".

OK, I'm lost here. Why is there a 1:1 correspondence between the two?

We can argue about "biggest" all day long but UTF-16 is a huge design failure because it made a huge chunk of the lower Unicode space unusable, thereby making better encodings like UTF-8 that could easily represent those code points less efficient. This layer-violating hack should have made it clear that UTF-16 was a bad idea from the start.

Then there is also the issue that technically there is no such thing as UTF-16, instead you need to distinguish UTF-16LE and UTF-16BE. Even though approximately no one uses the latter we still can't ignore it and have to prepend documents and strings with byte order markers (another wasted pair of code points for the sake of an encoding issue) which mean you can't even trivially concatenate them anymore.

Meanwhile UTF-8 is backwards compatible with ASCII, byte order independent, has tons of useful properties and didn't require any Unicode code point assignments to achieve that.

The only reason we have UTF-16 is because early adopters of Unicode bet on UCS-2 and were too cheap to correct their mistake properly when it became clear that two bytes wasn't going to be enough. It's a dirty hack to cover up a mistake that should have never existed.

> The only reason we have UTF-16 is because early adopters of Unicode bet on UCS-2 and were too cheap to correct their mistake properly

That's a strange way to characterize years of backwards compatibility to deal with

https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=10...

There are many OS interfaces that were deprecated after five years or even longer. It's been multiple times those five years since then and we'll likely have to deal with UTF-16 for much longer still. Having to provide backwards compatibility for UTF-16 interface doesn't mean they had to keep these as the defaults or provide new UTF-16 interfaces. In particular WIN32 already has 8-bit char interfaces that Microsoft could have easily added UTF-8 support to right then and re-blessed as the default. The decision not to do that was not a technical one but a political one.

This isn't "deprecate a few functions" -- it's basically an effort on par with migrating to Unicode in the first place.

I disagree you could just "easily" shove it into the "A" version of functions. Functions that accept UTF-8 could accept ASCII, but you can't just change the semantics of existing functions that emit text because it would blow up backwards compatibility. In a sense it is covariant but not contravariant.

And now, after you've gone through all of this effort: what was the actual payoff? And at what cost if maintaining compatibility with the other representations?

UTF-8 is superior simply because you can trivially choose to parse it as ascii and ignore all the weird foreign bytes.

> Now just UTF-16 and non '\n' newline types remaining to go

Also ISO 8601 (YYYY-MM-DD) should be the default date format.

We'll have to deal with it forever in network protocols. Thankfully that's rather walled off from most software.

As well as in a number of widely spread cryptographic algorithms (e.g. SHA-2), which use BE for historic reasons.

just call it 2-AHS and you're done :)

Good call out, I have just removed some #ifdef about endianness from my engine.

I have some places in some software where I assume little endian for simplicity, and I just leave in a static_assert(std::endian::native == std::endian::little) to let future me (or future someone else) know that a particular piece of code must be modified if it is ever to run on a not-little-endian machine.

Another good another call. Thank you.

In an ideal world you could just write endian-independent code (i.e. read byte by byte) and leave the compiler optimizer to sort it out. This has the benefit of also not tripping up any alignment restrictions.

Here's my most recent use case:

I have a relatively large array of uint16_t with highly repetitive (low entropy) data. I want to serialize that to disk, without wasting a lot of space. I run compress2 from zlib on the data when serializinsg it, and decompress it when deserializing. However, these files make sense to use between machines, so I have defined the file format to use compressed little endian 16-bit unsigned ints. Therefore, if you ever want to run this code on a big-endian machine, you need to add some code to first flip the bytes around before compressing, then flipping them back after decompressing.

You're right that when your code is iterating through data byte for byte, you can write it in an endian-agnostic way and let the optimizer take care of recognizing that your shifts and ORs can be replaced with a memcpy on little-endian systems. But it's not always that simple.

> It is amazing that big endian is almost dead.

I wish the same applied to written numbers in LTR scripts. Arithmetic operations would be a lot easier to do that way on paper or even mentally. I also wish that the world would settle on a sane date-time format like the ISO 8601 or RFC 3339 (both of which would reverse if my first wish is also granted).

> It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC.

I never really understood those non-8-bit bytes, especially the 7 bit byte. If you consider the multiplexer and demux/decoder circuits that are used heavily in CPUs, FPGAs and custom digital circuits, the only number that really makes sense is 8. It's what you get for a 3 bit selector code. The other nearby values being 4 and 16. Why did they go for 7 bits instead of 8? I assume that it was a design choice made long before I was even born. Does anybody know the rationale?

> I also wish that the world would settle on a sane date-time format like the ISO 8601

IIRC, in most countries the native format is D-M-Y (with varying separators), but some Asian countries use Y-M-D. Since those formats are easy to distinguish, that's no problem. That's why Y-M-D is spreading in Europe for official or technical documents.

There's mainly one country which messes things up...

YYYY-MM-DD is also the official date format in Canada, though it's not officially enforced, so outside of government documents you end up seeing a bit of all three formats all over the place. I've always used ISO 8601 and no one bats an eye, and it's convenient since YYYY-DD-MM isn't really a thing, so it can't be confused for anything else, unlike the other two formats.

YMD has caught on, I think, because it allows for the numbers to be "in order" (not mixed-endian) while still having the month before the day which matches the practice for speaking dates in (at least) the US and Canada.

The primary reason for YMD is that DDMMYYYY is ambiguous with MMDDYYYY

It is also sortable, which I think is the real advantage.

As ling as you pad to two characters!

I used to think this was really important, but what's the use case here?

If I'm writing a document for human consumption then why would I expect the dates to be sortable by a naive string sorting algorithm?

On the other hand, if it's data for computer consumption then just skip the complicated serialisation completely and dump the Unix timestamp as a decimal. Any modern data format would include the ability to label that as a timestamp data type. If you really want to be able to "read" the data file then just include another column with a human-formatted timestamp, but I can't imagine why in 2025 I would be manually reading through a data file like some ancient mathematician using a printed table of logarithms.

> If I'm writing a document for human consumption then why would I expect the dates to be sortable by a naive string sorting algorithm?

If you're naming a document for human consumption, having the files sorted by date easily without relying on modification date (which is changed by fixing a typo/etc...) is pretty neat

This is exactly it - file name is easy to control and sort on; creates date and modified date are (for most users) random and uncontrolled.

So you can't sort by name, author etc? One sort key? What year is it?!

I live in that country, and I am constantly messing up date forms. My brain always goes yyyy-mm-dd. If I write it out, September 1st, 2025, I get it in the “right” order. But otherwise, especially if I’m tired, it’s always in a sortable format.

8601, when used fully according to spec sucks. It makes today 20250902. It doesn't have seperators. And for adding a time it gets even less readable.

Its a serialization and machine communication format. And that makes me sad. Because YYYY-MM-DD is a great format, without a good name.

It does have a good name: RFC 3339. Unlike the ISO standard, that one mandates the "-" separators. Meanwhile it lets you substitute a space for the ugly "T" separator between date and time:

> NOTE: ISO 8601 defines date and time separated by "T". Applications using this syntax may choose, for the sake of readability, to specify a full-date and full-time separated by (say) a space character.

YYYY-MM-DD is ISO8601 extended format, YYYYMMDD is ISO8601 basic format (section 5.2.1.1 of ISO8601:2000(E)[1]). It's fully according to spec, and neither format takes precedence over the other.

[1] https://www.pvv.org/~nsaa/8601v2000.pdf

I always like the compromise of the M/D/M system popularised by the British documentary series Look Around You, e.g. "January the fourth of March".

I don't know that 7-bit bytes were ever used. Computer word sizes have historically been multiples of 6 or 8 bits, and while I can't say as to why particular values were chosen, I would hypothesize that multiples of 6 and 8 work well for representation in octal and hexadecimal respectively. For many of these early machines, sub-word addressability wasn't really a thing, so the question of 'byte' is somewhat academic.

For the representation of text of an alphabetic language, you need to hit 6 bits if your script doesn't have case and 7 bits if it does have case. ASCII ended up encoding English into 7 bits and EBCDIC chose 8 bits (as it's based on a binary-coded decimal scheme which packs a decimal digit into 4 bits). Early machines did choose to use the unused high bit of an ASCII character stored in 8 bits as a parity bit, but most machines have instead opted to extend the character repertoire in a variety of incompatible ways, which eventually led to Unicode.

On the DEC-10 the word size is 36 bits. There was (an option to include) a special set of instructions to enable any given byte size with bytes packed. Five 7-bit bytes per word, for example, with a wasted bit in each word.

I wouldn’t be surprised if other machines had something like this in hardware.

Could you use the extra bit for parity?

I don’t remember if there are instructions for putting any value of parity as such into the spare bit.

> For the representation of text of an alphabetic language, you need to hit 6 bits if your script doesn't have case

Only if you assume a 1:1 mapping. But e.g. the original Baudot code was 5-bit, with codes reserved to switch between letters and "everything else". When ASCII was designed, some people wanted to keep the same arrangement.

I wasn't asking about word sizes in particular, and had ASCII in mind. Nevertheless, your answer is in the right direction.

Quick note that parity was never used in "characters stored". It was only ever used in transmission, and checked/removed by hardware[1].

[1] Yes, I remember you could bit-bang a UART in software, but still the parity bit didn't escape the serial decoding routine.

I believe that 10- and 12-bit bytes were also attested in the early days. As for "why": the tradeoffs are different when you're at the scale that any computer was at in the 70s (and 60s), and while I can't speak to the specific reasons for such a choice, I do know that nobody was worrying about scaling up to billions of memory locations, and also using particular bit combinations to signal "special" values was a lot more common in older systems, so I imagine both were at play.

Computers never used 7-bit bytes similarly to how 5-bit bytes were uncommon, but both 6-bit and 8-bit bytes were common in their respective eras.

I was asking about ASCII encoding and not the word size. But this information is also useful. So apparently, people were representing both numbers and script codes (EBCDIC in particular) in packed decimal or octal at times. The standardization on 8 bits and adoption of raw binary representation seems to have come later.

Because character encodings were primarily designed around transmission over serial lines.

Agreed. But how does that affect how encoding are designed? I mean packed decimal vs octal vs full binary, etc?

There are a lot of computations where 256 is too small of a range but 65536 is overkill. When designers of early computers were working out how many digits of precision their calculations needed to have for their intended purpose 12 bits commonly ended up being a sweet spot.

When your RAM is vacuum tubes or magnetic core memory, you don't want 25% of it to go unused, just to round your word size up a power of two.

> There are a lot of computations where 256 is too small of a range but 65536 is overkill

wasnt this more to do with cost? they could do arbitrary precision code even back then. its not like they were only calculating numbers less than 65537, ignoring anything larger

In Britain the standard way to write a date has always been, e.g "12th March 2023” or 12/3/2022 for short. Don't think there's a standard for where to put the time, though, I can imagine it both before and after.

Doing numbers little-endian does make more sense. It's weird that we switch to RTL when doing arithmetic. Amusingly the Wikipedia page for Hindu-Arabic numeral system claims that their RTL scripts switch to LTR for numbers. Nope... the inventors of our numeral system used little-endian and we forgot to reverse it for our LTR scripts...

We in India use the same system for dates as you described, for obvious reasons. But I really don't like the pattern of switching directions multiple times when reading a date and time.

And as for numbers, perhaps it isn't too late to set it right once and for all. The French did that with the SI system after all.

> So apparently the original Hindu scripts were LTR

I can confirm. All Indian scripts are LTR (Though there are quite a few of them. I'm not aware of any exceptions). All of them seem to have evolved from an ancient and now extinct script named Brahmi. That one was LTR. It's unlikely to have switched direction any time during subsequent evolution into modern scripts.

> both of which would reverse if my first wish is also granted

But why? The brilliance of 8601/3339 is that string sorting is also correct datetime sorting.

> But why?

To get the little-endian ordering. The place values of digits increase from left to right - in the same direction as how we write literature (assuming LTR scripts), allowing us to do arithmetic operations (addition, multiplication, etc) in the same direction.

> The brilliance of 8601/3339 is that string sorting is also correct datetime sorting.

I hadn't thought about that. But it does reveal something interesting. In literature, we assign the highest significance to the left-most (first) letter - in the direction opposite to how we write. This needs a bit more contemplation.

> we assign the highest significance to the left-most (first) letter

Yes, we do that with everything, which is why little-endian numbers would be really inconsistent for humans.

7 bits was chosen to reduce transmission costs, not storage costs, because you send 12.5% less data. Also, because computers usually worked on 8-bit bytes, the 8th bit could be used as a parity bit, where extra reliability was needed.

> I also wish that the world would settle on a sane date-time format like the ISO 8601 or RFC 3339 (both of which would reverse if my first wish is also granted).

YYYY-MM-DD to me always feels like a timestamp, while when I want to write a date, I think of a name, (for me DD. MM. YYYY).

Big endian will stay around as long as IBM continues to put in the resources to provide first-class Linux support on s390x. Of course if you don’t expect your software to ever be run on s390x you can just assume little-endian, but that’s already been the case for the vast majority of software developers ever since Apple stopped supporting PowerPC.

> ...that’s already been the case for the vast majority of software developers ever since Apple stopped supporting PowerPC.

For better or worse, PowerPC is still quite entrenched in the industrial embedded space.

Right, and the vast majority of software developers aren't writing software intended to run on PowerPC industrial control boards.

"...with the CPU running in big endian mode."

Hey, you! You're supposed to be dead!

https://wiki.netbsd.org/ports/evbarm/