As you say, it can just show the sequence of bytes (plus some visual hints that they are escaped). It is not like nonsensical filenames with seem composed of random numbers are uncommon (e.g. look into your .git dir)
> How else does, for example, your shell find “ls” on your disk when you type those characters?
This is purely a presentation layer issue. The filesystem assumes nothing.
What you're suggesting would mean that a Korean user saving a file on their desktop and naming it in Korean would in turn see the filename turn into garbage, which, I hope we can all agree, is not a reasonable suggestion.
Creating 2 different systems carries risks: it has a learning curve, a support cost, and breaks everything that came before.
Perhaps, in a utopian green field, we redo a systems language, a portable systems API, and a common operating system with a compatibility layer to the old way on top.
Not changing impacts many more users (all the future billions) and costs even more: the learning/support/etc. is all there, and it's harder to learn about a buggy system since you have more papercuts
Also, it doesn't break everything, we're not living in a dystopia
2) Change what to what? There are multiple filesystems systems (and OSs) with different and sometimes incompatible filename encodings. Who picks the winner?
3) There is irony on dropping backward compat and requiring UTF-8, an econding whose claim to fame is backward compatibility with ASCII based systems (and not coincidentally was designed by UNIX Elder Ones)
1) so? That will be true for anything
2) sequence of bytes to human encoding. Who picked the loser??? This would similarly be some unidentifiable group of people. You can start by picking the winner for yourself, even at a conceptual level
3) no irony, sequence of bytes is not ASCII
The filesystem shouldn't arbitrate localization issues. It's the wrong place to do it in terms of performance, area-of-responsibility, and portability.
NUL-terminated UTF-8 in memory and size+data on disk is the best encoding.
Normalization is a multifaceted problem that needs to be handled at or before where the filename argument hits the standard library. Sometimes, normalization is NOT desirable and it is preferrable to normalize for comparison only to other strings and to faithfully store whatever was provided verbatim.
Side-effects and doing 737 MAX MCAS underneath users is the road to ruin.
Historically, you configured your OS to match (what you thought was) the encoding used on your disk, but that starts to break when you have network disks or when users exchange floppies.
In practice, I think about every Unix/Linux file system at least assumes ASCII encoding for the printable ASCII range. How else does, for example, your shell find “ls” on your disk when you type those characters?
So, not even in Unix/Linux is a file name “a bag of bytes”
I think file systems should enforce an encoding, and provide a way for users of the file system to find out what it is.
Apple’s HFS, for example, stored explicit ‘code page’ ID so that the Finder knew, for example, to use the Apple Cyrillic encoding to interpret the bytes describing file names on disk for that disk.
The modern way is to just say file names are valid Unicode strings, with UTF-8 being the popular choice. Unfortunately that introduces the normalization problem.