Not any that I'm aware of. You have your choice of programming languages where they got it wrong (Python 3, Ruby), languages which are incredibly nit-picky (Rust), languages which are full of footguns (C, C++), languages which pass on the issue (OCaml), languages which assume everything is UTF8 (Perl), languages which assume everything is UTF16 except when they forgot about planes and assume UCS-2 (Java, everything Windows), languages which are mad (APL), ...

Of these outcomes I like the choice to pick nits best. I felt the same way before I learned any Rust (ie I wrote C where I had to manually groom strings) so this went in my pile of things to like about Rust. Apparently Rust only chose relatively narrowly to have &str (the string slice type, which is guaranteed to be UTF-8 text) at all, rather than just &[u8] (a slice of bytes) everywhere and to my mind that's a pretty serious benefit.

The choice to have this in a language with the safe/ unsafe distinction works very nicely because in so many languages you'd have this promised UTF-8 type and then in practice everybody and their dog uses the unsafe assumed conversion because it's easier, but in Rust you're pulled up short because that conversion needs an unsafe block, your local style may require (and good practice certainly does) that you address this with a safety comment explaining why it's OK and... it just isn't, in most cases. So you write the safe conversion instead unless you really need not to. This is a really nice nudge, you can do the Wrong Thing™, but it's just easier not to.

Rust still has its warts. When dealing with an archive, say, you can find yourself needing to deal with Windows strings on Unix, or vice versa, but Rust only provides the string type for the platform you are running on. It could do with UnixString and WindowsString in addition to OsString.

MFC had a similar problem for years. It had a CString class which was ANSI or Unicode, depending on how your compiled your app, but moderately often you needed the other one, so it should have had a CStringA and CStringW too, with nice conversions between them.

OsString isn't "the native string type", it's a container for whatever was convenient for Rust's internals on that OS, there are convenience functions so it probably feels like it's "UnixString" or "WindowsString" but it's neither.

In most of these file format cases what you've got is &[u8] or &[u16] and maybe it's the NonZero variant instead, so I think it's fine to be explicit that's what is going on and maybe in the process remind you to check - is this data UTF16LE? UTF16 with a BOM? UCS2 with a nod and a wink? Just arbitrary 16-bit integers and good luck?

But like I said, I favoured "picking nits" long before I learned Rust, so mileage may vary.

IMHO Rust got OSString wrong – it indirectly promised that UTF-8 can always be cast to it without any copying or allocations, so on Windows it has to use WTF-8 rather than UCS2/UTF-16. Instead of being OS’s preferred string type, it’s merely a trick to preserve unpaired surrogates.

Huh. Where is the "indirect promise" ? Is there a (conventionally hinted as "free", so it's not good if there's one which isn't very cheap) as function? Like as_os_string or something?

It's because str implements AsRef<OsStr> [0]. The function signature promises that whenever you have a borrowed &'a str, the standard library can give you a borrowed &'a OsStr with the same data.

Since references can't have destructors (they don't own the data like an OsString does), it means that the standard library can't give you a newly-allocated string without leaking it. Since obviously it isn't going to do that, the &OsStr must instead just act as a view into the underlying &str. And the conversion can't enforce any extra restrictions on the input string without breaking backward compatibility.

The overall effect is that whatever format OsStr uses, it has to be a superset of UTF-8.

[0] https://doc.rust-lang.org/1.77.2/std/ffi/struct.OsStr.html#i...

Ah, an AsRef<OsStr>, yep that would do it. Thanks.

I wouldn't say it's necessarily wrong; it may be (accidental) foresight. Windows has added a UTF-8 code page, which means you can get as near enough as makes no difference full support with the A functions instead of the W functions.

That said, even now in 2024, it's not clear how much of a bet Windows is making on UTF-8 versus UTF-16.

I would be surprised if the UTF-8 support in Windows was anything deeper than the A functions creating a W string and calling W functions, which is what Rust is doing already.

The question is whether that's all they ever do, or whether gradually some parts of Windows begin to decide to be UTF-8 first and their W functions now translate the opposite way. This must be tempting for some networking stuff, where you're otherwise taking a perf hit compared to Linux.

The first time I had to deal with this was on Turbo Pascal for Windows, where we got C strings alongside Pascal ones.

Then when 32 bit came to be we had the other variations on top.

Got to love leaky abstractions.

I'm curious what you think python got wrong.

Your program will crash at runtime if you pass any string that isn't valid UTF-8 and you forgot to use bytes, and using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data.

It's a problem with Unix filenames where the encoding is just a convention. A Python program that doesn't take great care can crash on a parameter that takes a filename even if that filename is just passed to a function like 'open' so no sanitisation or conversion is necessary.

This is a very real problem in hivex, our Windows registry library, where the Python bindings don't really work well. The Windows registry is a hodge podge of random encodings, essentially whatever the program that wrote the registry key thought it was using at the time. When parsed through Python as a string this means you'll get unicode decoding errors all over the place.

Also more in this article: https://changelog.complete.org/archives/9938-the-python-unic...

You can tell .decode what to do with errors so it won't throw an exception, the default is 'strict'. I think python 3.6+ did a pretty good job with it overall.

Don't get me wrong it's still painful and annoying and bug prone, but the point is, it's encodings, it's always going to suck no matter what.

> It's a problem with Unix filenames where the encoding is just a convention

UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. They're sequences of bytes not containing 0x00 or 0x2F, but with no encoding.

Same for Windows registry keys. Don't mistake byte strings for text.

Text is a byte string with an encoding which describes which byte values are valid and which characters each byte (or sequence of bytes) corresponds to. If the encoding information is discarded, it stops being text and becomes just a byte string.

That's an important way of looking at it, and is correct as far as traditional UNIX operations like fopen go. But it isn't the whole story because many other operations require treating paths as text. For example, converting them to URIs, putting them in ZIP files, or looking up a file on a filesystem which internally stores filenames in UCS-2.

Threading both of these needles at once basically requires viewing paths as potentially-invalid encoded text.

> UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text.

But they nearly always also happen to be printable text. Treating them as printable text is soooo convenient 99.9% of the time.

Having them be almost printable text, but not quite, is an API design wart that is begging to be used incorrectly. It's the inverse of an affordance.

Sure, I can agree with that. Just because the API is bad doesn't mean you can treat it as though it's the API you wish it were.

OTOH, if "stop whining about how hard the system is to use correctly, just pay attention, be careful, and don't write bugs" worked, we wouldn't have issues like the one this thread is about, and no-one would have needed to invent Rust.

Oh, 100%. It's critical to wrap the crappy APIs into good ones. But that means you have to write the wrapper, and do so very carefully. Then hope you can get your wrapper adopted widely enough to be standardized, and have the old crappy API deprecated & removed (unlikely). In practice, the crappy API will be kept for backwards compatibility, and will forever remain a footgun. You have to be aware of it, even if only in a "here be dragons" manner.

The POSIX filename rules are a "here be dragons" sign. If you can avoid dealing with them, it's best to do so. If you can't avoid it, you'll need to parse your inputs into valid text, and fall back to a byte string handling path (or just fail with error) if they're not text. You can't safely just assume they're text & treat them as such, it doesn't end well.

To be honest, I disagree. Unix filenames are printable text, they're just text that the OS chooses not to enforce any validation on. Especially since the charset is implied by your choice of locale, which is a user decision, not a kernel decision. But we've transitioned into a world where pretty much all systems have settled on UTF-8 as the charset of choice, and the OS's refusal to even permit kernel options to forcibly validate filenames as UTF-8 is starting to look like a poor decision.

(IIRC, at least one of the BSDs has actually moved to forcing filenames to be UTF-8 and refusing path names that aren't UTF-8. Would only that Linux moved down that path as well so that we could be done with this farce.)

A filename consisting of nothing but ASCII Bell characters (0x07) is valid. Those are non-printable characters that (used to) make a sound from the PC's speaker. POSIX filenames can be sound, not text.

I'd agree it'd be nice if we could restrict filenames to valid UTF-8. But that's not the API that existing filesystems provide, nor what (most of) the existing OSes enforce.

Note that BEL(0x07) is valid UTF-8.

Kernel developers need to consider backwards compatibility. You won't want to see some users lose their data because they upgraded the kernel. Therefore it is very hard to "force" something.

Filesystem mount option, have utilities default it to on when creating new partitions.

>using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data.

How else would you decode a string without knowing its encoding? You can either guess (and risk invalid result/decode errors) or store this information somewhere. This is universal and true in every language. In most cases today people choose to guess utf-8.

How about:

    >>> 'abc' + 'def'[1]
    abce
    >>> b'abc' + 'def'[1]
    TypeError: can't concat int to bytes

It's been responsible for a lot of bugs in my code. They copied the "bytes is an array of integers" thing from Java. Big mistake. Python is not Java.

Them treating filenames as strings is another bug factory on Linux. Almost no one unit tests filenames with invalid encodings, so the result is a whole pile of python3 programs will fall when given perfectly valid input, whereas those same programs in python2 were fine.

It's a very odd outcome given python harks from Linux.

…languages which are mad..

Reference to “Celestial Emporium of Benevolent Knowledge”?

It was a sly dig at APL for using non-ASCII characters as regular operators. Actually I have no idea how those were implemented in the language. Presumably not as Unicode since APL predates Unicode by quite a considerable number of years. Does anyone know?

As a meta-joke I was also considering:

APL (j/k)

In modern APLs a character scalar is just a Unicode code point, which you might consider UTF-32. It's no trouble to work with. Although interfacing with things like filenames that can be invalid UTF-8 is a bit of a mess; Dyalog encodes these using the character range that is reserved for UTF-16 surrogates and therefore unused. If you know you want to work with a byte sequence instead, you can use Unicode characters 0-255, and an array of these will be optimized to use one byte per character in dialects that care about performance.

Java is pretty painless. Never had any issues with it.

Java doesn't read uniboms sanely. Heaven help you parsing mixed input files.

It must be very difficult to use one of the dozens of packages that let you detect and pick correct encoding to wrap a file stream before reading into a string.

Empirically yes! Nearly everything we tried would do a passable job on European texts, and misconvert short texts in less common character sets. Because the problem is ambiguous, we had the best luck with a library that allowed us to disable possible encodings. We thought we could just remove encodings that were not widely in use in the countries we served, and that mostly fixed it, but it still came up a couple times a year. Eventually we had a large enough corpus of sample documents that we could test any changes to the character set bitmap for impact to at least some of each better supported character set.

Just not my cup of tea though.

Not in C/C++ which is what makes you ask this question :)

Rust, C#, Java and Go are fairly straightforward in this regard.

`Encoding.{Name}.GetString/GetBytes`