Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by js2 on Wednesday July 18, 2018 @01:50PM
(#17560515)
It may not be clear till you read all the comments on the bug that the 9 which isn't matched is a FULLWIDTH DIGIT NINE.
If you're using, e.g., [a-fA-F0-9] in any locale other than C/POSIX, you're going to have a bad time.
This is going to cause problems, and it's a shame the developers note this, but then seem to wash their hands of it. Perhaps glibc should do what the programmer means, not what they say. It's obvious the regex above intends to validate hex digits, so glibc should do that, standards (and locale) be damned, unless the programmer explictly opts in to the technically correct behavior. The documentation even says:
> Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
If that range doesn't make sense except in the C/POSIX locale, then why interpret it in any other locale? Come on glibc... help us out here.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by zeroimpl on Thursday July 19, 2018 @10:42AM
(#17566727)
This suggests the only safe portable way to represent hex is one of these? [1234567890abcdefABCDEF] or [\dabcdefABCDEF]
That sucks. Fortunately I don't use C, but I'm worried about this finding its way into other languages...
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by js2 on Friday July 20, 2018 @11:48AM
(#17575987)
Javascript defines regex ranges in terms of UTF-16 code units w/no consideration for locale, so you can use [a-fA-F0-9] and it will work as expected.
The issue discussed in this bug is only relevant to POSIX regular expression ranges.
Perhaps you're lamenting that there are so many different flavors of regular expression. I agree. Just the other day I had to give up on using a regex in a CloudFormation template to validate input because I could not get it to work as documented.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by cpburns2009 on Wednesday July 18, 2018 @02:05PM
(#17560669)
This is bizarre. I thought [0-9] was supposed to match only 0-9 (the 10 digits), while \d was meant to match all digits including the various Unicode variations.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by jwilk on Thursday July 19, 2018 @04:48AM
(#17564748)
This is true for many regular expression dialects.
But this bug about POSIX regular expressions. POSIX doesn't define \d, and it defines ranges such as [0-9] only in the POSIX locale.
Historically, glibc considered such ranges in the context of locale's collation order. For example, in Estonian locale, [a-z] doesn't include the letters tuvwxy, because that's how Estonian alphabet works: https://en.wikipedia.org/wiki/Estonian_orthography#Alphabet
This (somewhat surprising) semantics of character ranges is not new, but the recent changes in glibc made it more spectacular.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by kazinator on Thursday July 19, 2018 @07:20PM
(#17571211)
\d is a Perlism.
Yes, [0-9] is supposed to be locale independent, which is related to the fact that ISO C requires the digit characters '0' through '9' to be consecutive.
The same is not true of [A-Z], which doesn't necessarily include just the upper-case English letters from ASCII. It is locale-specific, in fact.
The POSIX digit matcher is [[:digit:]]. It is related to the C isdigit function from <ctype.h>. Those functions have regex counterparts, like isalnum -> [[:alnum:]] and so on.
The way those functions have locale-dependent behavior is linked to how the corresponding regex classes have locale-dependent behavior.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by usr1106 on Thursday July 19, 2018 @01:46PM
(#17568727)
So where are glibc regexps used in the most typical practical cases?
At least grep does not use them (at least not in unmodified form) as a comment in the bug report notes.
I think changing such behavior is unacceptable, regardless of what the spec says. Here I would to propose the API stability promise of the Linux kernel: We don't break existing programs.
If somebody notes that the previous behavior is not correct according to the spec a new posixly correct mode can be introduced. But it should not magically become default.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by detaro on Thursday July 19, 2018 @10:45AM
(#17566749)
Does this "pass through" to the regex implementations in any other languages, or do those tend to implement their own parser or integrate different libraries?
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by js2 on Thursday July 19, 2018 @11:10AM
(#17567027)
Few languages use glibc for their regex implementation. PCRE is probably the most commonly used third-party regex lib, but it depends. Python and a bunch of languages have their own implementations.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by kazinator on Thursday July 19, 2018 @07:31PM
(#17571279)
> I guess we should all just force-set our C programs to the C locale.
Ah, but you are in luck: this is not required!
A C program begins execution in the "C" locale, regardless of the environment variables.
The locale variables do not affect a C program until it calls the "fuck this program function" called setlocale, with LC_ALL, and "". The empty string argument means "use the environment variable".
The problem is that many modern utilities in GNU/Linux land all make this setlocale call, in order to support internationalization. So then they are affected by whatever hell breaks loose.
Your own program is safe, if neither it, nor a third party library, calls setlocale, then your strcmp and isalpha and whatever else behave in good old naive manner.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by patrec on Wednesday July 18, 2018 @05:13PM
(#17562109)
This works till you want at least minimal UTF-8 support, such as when you're working with actual English text as opposed to human-readable computerese.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by kazinator on Thursday July 19, 2018 @07:26PM
(#17571249)
Sticking to the "C" locale will work even if you want UTF-8 support. You just roll your own UTF-8 support.
I built the TXR language entirely without any of the harmful garbage that is the ISO C/POSIX localization. It handles UTF-8 just fine.
The C localization stuff was developed too early, at a time when nobody had any real experience with localization. Before Unicode, before the Internet.
Before threads! How the do you set it up so that one thread runs in one locale and another in another? That's important if a global server is servicing two different requests simultaneously from users in two different locales. The idiotic C locale stuff relies on global variables.
You set some magic variables and, poof; numerous functions in your entire image change their behavior, whether they are working with internationalized data or not.
The setlocale function might as well be called fuck_my_program_please.
Re:Glibc: [0-9] matches ¼ ١ 2 〣 and others, but not 9 (and o...
by rwmj on Wednesday July 18, 2018 @02:46PM
(#17561027)
In all the scripted tests we write, one of the first lines will be:
export LANG=C
A similar construct is also used in most places where we call sort (except where we're actually sorting strings of text that will be read by an end user), eg:
If you're using, e.g., [a-fA-F0-9] in any locale other than C/POSIX, you're going to have a bad time.
This is going to cause problems, and it's a shame the developers note this, but then seem to wash their hands of it. Perhaps glibc should do what the programmer means, not what they say. It's obvious the regex above intends to validate hex digits, so glibc should do that, standards (and locale) be damned, unless the programmer explictly opts in to the technically correct behavior. The documentation even says:
> Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
If that range doesn't make sense except in the C/POSIX locale, then why interpret it in any other locale? Come on glibc... help us out here.
$0.02.
The issue discussed in this bug is only relevant to POSIX regular expression ranges.
Perhaps you're lamenting that there are so many different flavors of regular expression. I agree. Just the other day I had to give up on using a regex in a CloudFormation template to validate input because I could not get it to work as documented.
See https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c28 and https://www.sourceware.org/ml/libc-alpha/2018-07/msg00620.ht...
But this bug about POSIX regular expressions. POSIX doesn't define \d, and it defines ranges such as [0-9] only in the POSIX locale.
Historically, glibc considered such ranges in the context of locale's collation order. For example, in Estonian locale, [a-z] doesn't include the letters tuvwxy, because that's how Estonian alphabet works: https://en.wikipedia.org/wiki/Estonian_orthography#Alphabet
This (somewhat surprising) semantics of character ranges is not new, but the recent changes in glibc made it more spectacular.
Yes, [0-9] is supposed to be locale independent, which is related to the fact that ISO C requires the digit characters '0' through '9' to be consecutive.
The same is not true of [A-Z], which doesn't necessarily include just the upper-case English letters from ASCII. It is locale-specific, in fact.
The POSIX digit matcher is [[:digit:]]. It is related to the C isdigit function from <ctype.h>. Those functions have regex counterparts, like isalnum -> [[:alnum:]] and so on.
The way those functions have locale-dependent behavior is linked to how the corresponding regex classes have locale-dependent behavior.
With ignorecase, !\W didn't match 'k' and 's' (only!)
https://bugs.ruby-lang.org/issues/4044
At least grep does not use them (at least not in unmodified form) as a comment in the bug report notes.
I think changing such behavior is unacceptable, regardless of what the spec says. Here I would to propose the API stability promise of the Linux kernel: We don't break existing programs.
If somebody notes that the previous behavior is not correct according to the spec a new posixly correct mode can be introduced. But it should not magically become default.
Ah, but you are in luck: this is not required!
A C program begins execution in the "C" locale, regardless of the environment variables.
The locale variables do not affect a C program until it calls the "fuck this program function" called setlocale, with LC_ALL, and "". The empty string argument means "use the environment variable".
The problem is that many modern utilities in GNU/Linux land all make this setlocale call, in order to support internationalization. So then they are affected by whatever hell breaks loose.
Your own program is safe, if neither it, nor a third party library, calls setlocale, then your strcmp and isalpha and whatever else behave in good old naive manner.
I built the TXR language entirely without any of the harmful garbage that is the ISO C/POSIX localization. It handles UTF-8 just fine.
The C localization stuff was developed too early, at a time when nobody had any real experience with localization. Before Unicode, before the Internet.
Before threads! How the do you set it up so that one thread runs in one locale and another in another? That's important if a global server is servicing two different requests simultaneously from users in two different locales. The idiotic C locale stuff relies on global variables.
You set some magic variables and, poof; numerous functions in your entire image change their behavior, whether they are working with internationalized data or not.
The setlocale function might as well be called fuck_my_program_please.
You should set LC_ALL=C instead.
So just set LC_ALL and be done with it. It's bulletproof.