Breaking our Latin-1 assumptions (2017)

Re:Breaking our Latin-1 assumptions (2017) by jfk13 on Sunday June 19, 2022 @06:30AM (#31798052)

Unfortunately, your "text safe" definition appears to exclude text in languages such as Persian (among others), where Zero Width Non-Joiner is required to write some words correctly.

[ Parent | on HN ]

Re:Breaking our Latin-1 assumptions (2017) by kstenerud on Sunday June 19, 2022 @07:25AM (#31798320)

Oh damn good catch! Adding category Cf.

[ Parent | on HN ]

Re:Breaking our Latin-1 assumptions (2017) by jfk13 on Sunday June 19, 2022 @01:06PM (#31801175)

Thanks.... though note that Cf also includes things like the bidi directional-override code points, which you might still prefer to exclude.

[ Parent | on HN ]

Re:Breaking our Latin-1 assumptions (2017) by kstenerud on Sunday June 19, 2022 @01:31PM (#31801440)

Yup, I've removed a bunch of Cf codes (it really is a grab bag!), but I'm not sure if removing BIDI could be done without breaking some languages?

https://github.com/kstenerud/concise-encoding/blob/master/ce...

[ Parent | on HN ]