Yeah, I recall doing
import codecs
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stderr = codecs.getwriter('utf-8')(sys.stderr)
to fix Unicode woes in Python 2, which opened another can of worms. Not fun.> Python 3 forces UTF-8 everywhere
If only. My problems with Python 3 were due to things like crashing when piping stdout to a file because for some reason the Python-chosen encoding for that was ASCII-only.
Another was discovering that the Python 3 default encoding for text files on Windows was CP-1252, so it would crash outputting any non-English text to a file.
It's a little better now, but the defaults were awful when I first started using Python 3. I expected it to be Unicode by default. It was not.
Yes, I omitted that my Python 2 (mostly Django or bottle) ran on Linux with UTF-8. I suspect your piping case could be due to either the OS not using UTF-8 as locale or the application behind the pipe ignoring the OS' locale.
These encoding problems will probably stay with us for quite a while, regardless of the language we write our applications in.
Just to elaborate, on a fresh Windows 7 install with en_CA locale, this was the behaviour:
python3 -c 'print("é")'
é
python3 -c 'print("é")' > out.txt
(UnicodeEncodeError) 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)
I felt that was unacceptable behaviour for my program, but it was beyond my ability to fix. Though, I think this has been addressed in more recent Python releases.I thought you might have fun with this -- try running my example here: https://news.ycombinator.com/item?id=21842471
Perhaps the main difference is the native language of the developer and his/her clients/customers: If you and your customers are native english speakers, encoding problems were dramatically less probable before because all letters in use were ASCII anyway - and the UTF-8 positions of ASCII characters stayed the same (e.g. 'A' is still 0x41)- so buggy encoding didn't tend to break things for them. So even older Windows versions encoding filenames in latin1 etc. were never a problem to begin with - something that was always a hazzle for me (natively German speaking, with umlauts etc.).
Python 3 forces UTF-8 everywhere, which solved oodles of quirks for everyone but natively english speaking people. And thankfully encoding bugs now tend to break things mostly on startup, not just if a customer uploads a funky file from an old OS, two months into deployment.