> In an ideal world, you would know what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant.
Twitter is in a known encoding, UTF-8. Most of ftfy's examples come from Twitter. ftfy is not redundant.
When ftfy gets the input "#╨┐╤Ç╨╨╨╕╨╗╤î╨╜╨╛╨╡╨┐╨╕╤é╨╨╜╨╕╨╡", it's not because this tweet was somehow in a different encoding, it's because the bot that tweeted it literally tweeted "#╨┐╤Ç╨╨╨╕╨╗╤î╨╜╨╛╨╡╨┐╨╕╤é╨╨╜╨╕╨╡", in UTF-8, due to its own problems. So you decode the text that was tweeted from UTF-8, and then you start fixing it.
I still think you're thinking of chardet.
> If instead, as is often the case in the real world, the coding is unknown...
...then you will need to detect its encoding somehow. By now ftfy is a library for Python 3 only. If you try to pass bytes into the ftfy function, the Python language itself will stop you.
Are you hypothesizing that everyone dealing with unmarked bytes is passing them through a chain of chardet and ftfy, and blaming ftfy for all the problems that would result?
Incidentally, I do machine learning. (That's why I had to make ftfy, after all.) I have tried many machine learning solutions. They do not come close to ftfy's heuristics, which are designed to have extremely low false positive rates that are not attainable by ML. If you want one false positive per billion inputs... you're going to need like a quadrillion inputs, or you're going to need a lovingly hand-tuned heuristic.
If it isn’t an encoding guesser, what does it do that "".decode("encoding") doesn’t do?
FTFY answers the question: What horrifying sequence of encode/decode transforms could output this sequence of bytes in UTF-8 that, when correctly decoded as UTF-8, still results in total gibberish?
In other words...
The problem fixed by an encoding guesser:
1. I encode my text with something that's not UTF-8-compatible.
2. I lie to you and say it's UTF-8.
3. You decode it as UTF-8 and get nonsense. What the heck?
4. A guesser tells you what encoding I actually used.
5. You decode it from the guessed encoding and get text.
----
The problem fixed by FTFY:1. I encode string S with non-UTF-8 codec C.
2. I lie that it's UTF-8.
3. Someone decodes it as UTF-8. It's full of garbage, but they don't care.
4. They encode that sequence of nonsense symbols, not the original text, as UTF-8. Let's charitably name this "encoding" C'.
5. They say: Here teddyh, take this nice UTF-8.
6. You decode it as UTF-8. What the heck?
7. Is it ISO-8859? Some version of windows-X? Nope. It's UTF-8 carrying C', a non-encoding someone's broken algorithm made up on the spot. There's no decoder that can turn your UTF-8 back into the symbols of S, because the text you got was already garbage.
8. FTFY figures out what sequence of mismatched encode/decode steps generates text in C' and does the inverse, giving you back C^-1( C'^-1( C'( C( S )))) = S.
The underlying ftfy library was previously discussed three years ago (https://news.ycombinator.com/item?id=8187418) and my comments at the time are still relevant:
I can’t help but think that this [library] merely gives people the excuse they need for not understanding this “Things-that-are-not-ASCII” problem. Using this library is a desperate attempt to have a just-fix-it function, but it can never cover all cases, and will inevitably corrupt data. To use this library is to remain an ASCII neanderthal, ignorant of the modern world and the difference of text, bytes and encodings.
Let me explain in some detail why this library is not a good thing:
In an ideal world, you would know what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant.
If instead, as is often the case in the real world, the coding is unknown, there exists the question of how to resolve the numerous ambiguities which result. A library such as this would have to guess what encoding to use in each specific instance, and the choices it ideally should make are extremely dependent on the circumstances and even the immediate context. As it is, the library is hard-coded with some specific algorithms to choose some encodings over others, and if those assumptions does not match your use case exactly, the library will corrupt your data.
A much better solution would perhaps involve a machine learning solution to the problem, and having the library be trained to deduce the probable encodings from a large set of example data from each user’s individual use case. Even these will occasionally be wrong, but at least it would be the best we could do with unknown encodings without resorting to manual processing.
However, a one-size-fits-all “solution” such as this is merely giving people a further excuse to keep not caring about encodings, to pretend that encodings can be “detected”, and that there exists such a thing as “plain text”.
[…]
I have […] two main arguments:
1. Due to its simplicity for a large group of naïve users, the library will likely be prone to over- and misuse. Since the library uses guessing as its method of decoding, and by definition a guess may be wrong, this will lead to some unnecessary data corruption in situations where use of this library (and the resulting data corruption) was not actually needed.
2. The library uses a one-size-fits-all model in the area of guessing encodings and language. This has historically proven to be less than a good idea, since different users in different situations use different data and encodings, and [the] library’s algorithm will not fit all situations equally well. I [suggest] that a more tunable and customizable approach would indeed be the best one could do in the cases where the encoding is actually not known. (This minor complexity in use of the library would also have the benefit of discouraging overuse in unwarranted situations, thus also resolving the first point, above.)