Re:Ftfy – fix Unicode that's broken in various ways
by cosmie on Tuesday January 09, 2018 @03:54PM
(#16109944)
It's probably a more mundane issue related to either older versions of HTML (which did not support UTF8[1]), or issues with an older database that stored things as latin1 or ASCII by default, or issues with an older programming language that doesn't seamlessly support non-latin1 or non-ascii characters without deliberately doing so (like Python 2).
That said, I do agree it's not something that should be used without thought. I mentioned that my usage generally involves knowledge that the data set has no valid reason to contain higher code points, and usually involves usage of Unidecode[2] to convert higher code points into ASCII equivalents (which in many cases strips out contextual knowledge, but is sometimes an acceptable trade off for stability and predictability of the data sent downstream).
That said, I do agree it's not something that should be used without thought. I mentioned that my usage generally involves knowledge that the data set has no valid reason to contain higher code points, and usually involves usage of Unidecode[2] to convert higher code points into ASCII equivalents (which in many cases strips out contextual knowledge, but is sometimes an acceptable trade off for stability and predictability of the data sent downstream).
[1] https://www.w3schools.com/html/html_charset.asp [2] https://pypi.python.org/pypi/Unidecode