Re:So you think you know what a number is
by evincarofautumn on Friday April 28, 2017 @05:07PM
(#14223050)
DB B2, etc. are the UTF-8 encodings of U+06F2, etc. So Python is seeing mojibake: U+00DB (Û), U+00B2 (²), etc. which are not digits. Well, one of them kinda is, but it’s No (“Number, other”), not Nd (“Number, decimal digit”).
[ Parent
| on HN ]
Yeah I get that, but why is that happening?
Re:So you think you know what a number is
by evincarofautumn on Friday April 28, 2017 @09:02PM
(#14224377)
I’d guess because CPython is assuming the input to -c is ISO-8859-1 (Latin-1) when it decodes it using Py_DecodeLocale():
main()
…
setlocale(LC_ALL, "")
…
argv_copy[i] = Py_DecodeLocale(argv[i], NULL)
…
mbstowcs() or mbrtowc()
…
setlocale(LC_ALL, oldloc)
…
Py_Main(argc, argv_copy)
While the REPL’s encoding (sys.stdin.encoding) is set to UTF-8 due to LANG/LC_CTYPE settings. You can get the same error when invoking the REPL as: LANG="en_US.iso8859-1" python2.7
So the shell isn’t doing anything to the text—it’s providing UTF-8 bytes in both cases, it’s just that Python is interpreting them differently.