News for hackers. Stuff that matters.
Re:An OCR cliche: Into his/her anus (2009) by superkuh on Sunday November 04, 2018 @08:32AM (#18375762)
Yup. I have to manually detect and correct for all the possible ligatures in all possible unicode in my text to speech pre-processor scripts. I hate them.
Re:An OCR cliche: Into his/her anus (2009) by dunham on Sunday November 04, 2018 @09:19AM (#18375957)
If you have a Unicode library available, you might try asking it to convert the text to NFKD or NFKC normalization form. This will take apart ligatures (the former will also take apart accented characters).
Re:An OCR cliche: Into his/her anus (2009) by ahazred8ta on Sunday November 04, 2018 @09:14AM (#18375935)
"this gives us efficient space-time trade-offs" :-(
Re:An OCR cliche: Into his/her anus (2009) by dotancohen on Sunday November 04, 2018 @01:13PM (#18377200)
Those are HTML entities. Most modern programming languages come with tools to decode this, e.g. in python:

    text = urllib.parse.unquote(text)
Re:An OCR cliche: Into his/her anus (2009) by jwilk on Sunday November 04, 2018 @03:12PM (#18377849)
urllib.parse.unquote() is unrelated to HTML. It undoes URL-encoding:

https://docs.python.org/3/library/urllib.parse.html#urllib.p...

In Python ≥ 3.4, you can use html.unescape() to decode HTML entities:

https://docs.python.org/3/library/html.html#html.unescape

Re:An OCR cliche: Into his/her anus (2009) by dotancohen on Monday November 05, 2018 @06:51AM (#18381574)
You are 100% correct. I mixed the two encodings up. Thanks.