I actually view this as a feature. Encoding/decoding of data should be an application level thing, not an OS-level thing. As far as the OS is concerned, data should be bytes.
(Of course, it is true that, since POSIX defines a terminal spec, it has to at least specify how bytes are mapped to characters that print on the terminal. But I would rather see that removed altogether, so a terminal becomes just another application, than have an OS try to muck about with encodings.)
That's because text encoding and decoding is a mess. Operating systems doing it doesn't make it any less of a mess; it just inserts the mess deeper into everything. For example, look at all the quirks and edge cases in file name handling between different OS's, simply because nobody is willing to just admit that to the OS, file names should be sequences of bytes, which are easy to share between machines running different OS's.
The basic issue is that text encoding and decoding exists because bytes have meanings. But unless/until we invent artificial intelligence, computers can't deal with meanings (because the meanings are not simple computable functions of the bytes). And OS's, particularly, should not even try. Applications might have to try, but the cost if they get it wrong is much less.
I think it's naive to think Operating Systems aren't going to fragment in order to offer "features" (and lockin), and then papering over all that fragmentation has to happen in the application anyway.
unless there's a standard, and if there's a standard the application itself can deal with it.
I have no problem with this as long as the metadata itself is just additional bytes. But if the metadata needs to be decoded in order to figure out how to decode it, we have a problem... :-)
You're missing my point. The OS should provide a way to store bytes. That's it. The meaning of the bytes is up to the application. If, to the application, the bytes represent text with a certain encoding, then it's up to the application to figure out how to translate the bytes, possibly using other stored bytes to decide. The OS doesn't need to get involved in any of this.
> it does nothing to help applications ensure that they correctly interpret data created by a different application
This is already a solved problem, and it isn't solved by OS's. It's solved by standards. For example, every web browser constantly has to correctly interpret data created by a different application. It can do so because HTML, CSS, JS, etc. are all standards that define how the bytes sent from the server to the client are to be interpreted. The browser doesn't even have to care what OS it's running on; all the OS is doing is giving it network sockets and a place for local data storage.
> If you're going to give your platform official APIs to address the very real problem of handling string encodings
If "platform" means "OS", then no, I'm not. If "platform" means "application framework", then sure, but an application framework is not the same thing as an OS. The fact that many OS's insist on also being application frameworks does not make the two things the same.
As I said originally, we tried that, and it doesn't work. Even within the context of a single locale, .NET applications will happily emit UTF-16 to be consumed by a Python script expecting all strings to be UTF-8, and with only byte-oriented APIs there's no side channel to convey that there's a mismatch that needs to be reconciled. Extending this problem from file and pipe contents to filenames is moving in the wrong direction. Operating systems absolutely should get involved in helping applications safely and usefully exchange information; that doesn't destroy the concept of an application framework, it just means that your OS is more than a hypervisor.
If the OS knows the type of those bytes, it can do things like implement global garbage collection, intelligent caching, intelligent snapshotting &c. It can also enforce invariants across all user code.
Hardware encode/decode often requires DMA capabilities. There are many optimizations that kernel mode can bring.
Of text?
- User level security, instead of app level or task level, or whatever.
- Aged IPC primitives that assume too much to the point that modern hardware has to be built around it (and slower because of that).
- Added later, non-core network support, leading to bad integration.
- Added later, non-core, or sometimes never added encryption support, leading to bad integration.
- (Correct) IO is ridiculously non-portable and painful in so many ways that it isn't even funny anymore
- Locks are ridiculously non-portable and painful to the point where you're better off just using "mkdir"
- POSIX is stuck in the "everything is bytes and we slap an encoding on it some of the time" era thinking. This makes it painful and hard to implement proper text handling in many instances. This also leads to a lot of bad behaviours.
- Memory management is IMHO lacking from a user space perspective. For example, it's practically impossible to implement a cooperative memory cache on this. To the best of my knowledge no OS has the necessary interfaces, though.
- ioctl as you mentioned
- SysV/POSIX IPC is so bad that no one ever bothered actually using it for anything
- Personally I think it's a misleading API (conceptually, see above, the text example for example) almost to the point of deceptiveness. It's very easy to write correct looking programs that behave far from intended, especially in edge cases. IMHO code using it is practically unreviewable in everything but the most trivial cases. Non-portability is practically guaranteed, you have to test every platform. Portable code usually turns out to be quite ugly due to platform deficiencies and minor API incompatibilities.