Speaking to the Haskell part:

An ExceptT or Maybe monad for handling encoding errors feels a lot like throwing exceptions, although they are less disruptive than exceptions. I'd probably represent a decoder as a function with type ByteString -> ErrorT ParseError m Text, which is neither an ADT or Typeclass. It's a 3rd solution. Either that or an Attoparsec parser, which is probably equivalent. An encoder seems like it shouldn't fail at all, but if it eventually forks out to one of the C locale functions I can see it throwing errors too.

Meanwhile, in the real world, Data.Text.Encoding uses a 4th solution implementing decodeUtf8With and encodeUtf8 that ultimately represents the UTF-8 encoding as a pair of FFI functions with these signatures:

  foreign import ccall unsafe "_hs_text_decode_utf8"   c_decode_utf8
      :: MutableByteArray# s -> Ptr CSize
      -> Ptr Word8 -> Ptr Word8 -> IO (Ptr Word8)

and this one:

  foreign import ccall unsafe "_hs_text_encode_utf8" c_encode_utf8
      :: Ptr (Ptr Word8) -> ByteArray# -> CSize -> CSize -> IO ()

text-icu also ultimately represents an encoding as an opaque pointer returned by the ICU library, and works in the IO monad. So it too could fail in similar ways. Errors throw an exception of type ICUError, which the caller can catch using the 'catch' function from Control.Exception.

The encoding library does use typeclasses like you suggest, but I'm not sure anybody uses it. Sometimes people drop in #haskell and complain about that library, and the response is usually "don't use that; use the one in Data.Text.Encoding instead".

I don't use Rust, but if the language is at all practical, I imagine they shuttle their equivalent of pointers and bytestrings around and depend on foreign C libraries and locales just the same. Probably they don't want to change the core library every time the Unicode Consortium publishes a new encoding scheme, so I can't imagine them exposing only a closed type.

So, looking purely at the signature and comparing it to examples from a language you suggested, it doesn't appear to be poorly designed at all. It's exactly what I would expect and want in any language, and the library consensus seems to agree. I think you're just imagining the grass being greener on the other side.

In Rust, our main two string types are String and &str, which are both UTF-8 encoded. For interoperability with other things, we have additional types that you can convert to/from. http://andrewbrinker.github.io/blog/2016/03/27/string-types-... is a recent overview in a blog post.

You still run into the problem with other functions, though. For example, in Haskell, 'tail' is a partial function - it's undefined if the list is nil. If, in your code, you write:

  xs = if p x then concat [[x, "bar"], foos] else "baz" : foos
  ys = tail xs

Then you know that the call to tail is not going to fail in your code, because the input has a guaranteed minimum length of either 1 or 2. You can't make that same guarantee about tail in isolation, though.

Sometimes you need also a user provided encoding (think of editors). In that case, the exception makes sense. Haskell or Rust would need to provide an extra API for this case. But generally you are right, stronger type checking would be preferable. Anyway, I dislike API's which take a String but only support a strongly limited subset of these. In that case, a dedicated type suits much better.

That's what typeclasses are for.

Rather than “poorly designed”, I'd say “reflects a limitation of the language”. You can't blame libraries for language defects.