Re:Unicode Utilities: Confusables
by hilbert42 on Tuesday August 23, 2022 @08:45AM
(#32564077)
FWIW, this example is really, really far down on the list of things you'll be going to typographic and/or semantic hell for. It doesn't impede communication,...
Totally agree, it's just that on occasions I've seen others raise the matter as an excellent example of how complex Unicode can be. Using a well-known, well used example clearly illustrates the point.
However, the matter of data integrity and machine learning is another matter altogether as meanings can easily be misinterpreted if wrong characters are used. One of the biggest problems of course is the transliterations of scripts both ancient and modern. This is a serious issue for historians and archivists and it's the reason for why Unicode makes such distinctions.
Considering these serious limitations, it's little wonder why OCR is not used as much as it ought to be. Take any text PDF on the Internet Archive that's derived from an image scan of a book and you'll find the error rate to be enormous. Clearly, these errors are why the IA offers books in both text and 'raw' image forms. Similarly, many newspaper publishers have usergroups of readers who correct scans of historical newspapers.
Obviously, this is very time consuming and somewhat defeats the purpose of OCR, it still however is a more accurate and much faster process than retyping the text.
The only real longterm solution is AI/machine learning and even then it won't be perfect. Statistics (hit/error rates for any given text) will come into play. Texts that are converted with, say, a deemed accuracy of 99.99% [my guesstimate] would be exempt from further correction but still marked with '99.99%' in the attached metadata - here the metadata is everything. Others will be flagged for correction with the potential errors flagged.
This needs new conversion format rules for OCR, as I see it, both metadata and the original scans should always be embedded within the 'converted' file. As far as I'm aware whilst work has progressed along these lines we're still a long way from a practical working solution.
Totally agree, it's just that on occasions I've seen others raise the matter as an excellent example of how complex Unicode can be. Using a well-known, well used example clearly illustrates the point.
However, the matter of data integrity and machine learning is another matter altogether as meanings can easily be misinterpreted if wrong characters are used. One of the biggest problems of course is the transliterations of scripts both ancient and modern. This is a serious issue for historians and archivists and it's the reason for why Unicode makes such distinctions.
Considering these serious limitations, it's little wonder why OCR is not used as much as it ought to be. Take any text PDF on the Internet Archive that's derived from an image scan of a book and you'll find the error rate to be enormous. Clearly, these errors are why the IA offers books in both text and 'raw' image forms. Similarly, many newspaper publishers have usergroups of readers who correct scans of historical newspapers.
Obviously, this is very time consuming and somewhat defeats the purpose of OCR, it still however is a more accurate and much faster process than retyping the text.
The only real longterm solution is AI/machine learning and even then it won't be perfect. Statistics (hit/error rates for any given text) will come into play. Texts that are converted with, say, a deemed accuracy of 99.99% [my guesstimate] would be exempt from further correction but still marked with '99.99%' in the attached metadata - here the metadata is everything. Others will be flagged for correction with the potential errors flagged.
This needs new conversion format rules for OCR, as I see it, both metadata and the original scans should always be embedded within the 'converted' file. As far as I'm aware whilst work has progressed along these lines we're still a long way from a practical working solution.