News for hackers. Stuff that matters.
Re:Ftfy – fix Unicode that's broken in various ways by rspeer on Tuesday January 09, 2018 @08:19AM (#16105920)
But if Chromium ran ftfy on all text, then you wouldn't be able to read the examples on the page, or in your post :(

(In general, I will claim that the only false positives you're likely to encounter using ftfy are test cases for ftfy!)

Re:Ftfy – fix Unicode that's broken in various ways by exikyut on Tuesday January 09, 2018 @10:14PM (#16112307)
Very good point.

This is an awesome library. My real question is... how on earth does it figure out the text is "okay", that the demangling process is done?

Re:Ftfy – fix Unicode that's broken in various ways by rspeer on Wednesday January 10, 2018 @12:20AM (#16112731)
Each de-mangling step decreases a "cost" metric on the text, based on its length plus the number of unusual combinations of characters. It never really decides that text is "okay", but when there's no step it can take that decreases the cost, it's done.

This is an imperfect greedy strategy, incidentally. If it takes multiple steps to fix some text, it's possible that the first step it needs to take is not the one that decreases the cost as much as possible, that it has to go through some awful-looking intermediate state so that everything falls into place for the next step. This is rare, though. I don't think I could come up with an example.

Re:Ftfy – fix Unicode that's broken in various ways by exikyut on Friday January 12, 2018 @03:07AM (#16131271)
oooooh. Very interesting.

And I think I understand what you mean.