News for hackers. Stuff that matters.
Re:The Fundamental Problem in Python 3 by bildung on Friday December 20, 2019 @02:31AM (#21842296)
Interestingly, I had exactly the opposite situation: I always had codec problems when using Python 2, and they were all solved with Python 3.

Perhaps the main difference is the native language of the developer and his/her clients/customers: If you and your customers are native english speakers, encoding problems were dramatically less probable before because all letters in use were ASCII anyway - and the UTF-8 positions of ASCII characters stayed the same (e.g. 'A' is still 0x41)- so buggy encoding didn't tend to break things for them. So even older Windows versions encoding filenames in latin1 etc. were never a problem to begin with - something that was always a hazzle for me (natively German speaking, with umlauts etc.).

Python 3 forces UTF-8 everywhere, which solved oodles of quirks for everyone but natively english speaking people. And thankfully encoding bugs now tend to break things mostly on startup, not just if a customer uploads a funky file from an old OS, two months into deployment.

Re:The Fundamental Problem in Python 3 by oefrha on Friday December 20, 2019 @02:47AM (#21842364)
Yeah, I recall doing

  import codecs

  sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
  sys.stderr = codecs.getwriter('utf-8')(sys.stderr)
to fix Unicode woes in Python 2, which opened another can of worms. Not fun.
Re:The Fundamental Problem in Python 3 by slavik81 on Friday December 20, 2019 @02:45AM (#21842346)
> Python 3 forces UTF-8 everywhere

If only. My problems with Python 3 were due to things like crashing when piping stdout to a file because for some reason the Python-chosen encoding for that was ASCII-only.

Another was discovering that the Python 3 default encoding for text files on Windows was CP-1252, so it would crash outputting any non-English text to a file.

It's a little better now, but the defaults were awful when I first started using Python 3. I expected it to be Unicode by default. It was not.

Re:The Fundamental Problem in Python 3 by bildung on Friday December 20, 2019 @02:57AM (#21842394)
Yes, I omitted that my Python 2 (mostly Django or bottle) ran on Linux with UTF-8. I suspect your piping case could be due to either the OS not using UTF-8 as locale or the application behind the pipe ignoring the OS' locale.

These encoding problems will probably stay with us for quite a while, regardless of the language we write our applications in.

Re:The Fundamental Problem in Python 3 by slavik81 on Friday December 20, 2019 @03:18AM (#21842465)
Just to elaborate, on a fresh Windows 7 install with en_CA locale, this was the behaviour:

  python3 -c 'print("é")'
  é
  python3 -c 'print("é")' > out.txt
  (UnicodeEncodeError) 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)
I felt that was unacceptable behaviour for my program, but it was beyond my ability to fix. Though, I think this has been addressed in more recent Python releases.
Re:The Fundamental Problem in Python 3 by mehrdadn on Friday December 20, 2019 @03:31AM (#21842527)
I thought you might have fun with this -- try running my example here: https://news.ycombinator.com/item?id=21842471