The problem with that view is that there are things for which you can not be correct, and there are no encoding issues because there is no encoding (or if there is one it does not map to proper unicode):
* UNIX files and paths have no encoding, they're just bags of bytes, with specific bytes (not codepoints, not characters, bytes) having specific meaning
* Windows file and path names are sequences of UTF-16 code units but not actually UTF-16 (they can and do contain unpaired surrogates), as above with specific code units (again not codepoints or characters) having specific meaning
These are issues you will encounter on user systems, there is no "forcing you to be correct". A non-unicode path is not incorrect. On many systems it just is. OSX is one of the few systems where a non-unicode path is actually incorrect, and that means you will not encounter one as input so you have no reason to handle this issue at all.
> Only his third bit of advice isn’t wrong and treating it as something unusual shows the problem: the only safe way to handle text
That's where you fail, and to an extent so does Python: some text-like things are not actually text. Path names famously is one of this case. You're trying to hammer the square peg of path names in the round hole of unicode.
Python tools for backups are my worst terror beacuse of that - they kept destroying data of our clients because they dared (gasp!) name files with characters from their own language or do unthinkable things, like create documents titled "CV - <name with non-unicode characters>.docx".
The fact that Python3 at least tries to make programmers not destroy data as soon as you type in a foreign name (which happens even in USA) is a good thing.
You can have badly written tools in any language. There are even functions to get file paths as bytes (e.g. [os.getcwdb](https://docs.python.org/3/library/os.html#os.getcwdb), it's just most people don't use them because it's rare-ish to see horribly broken filenames and not convenient.
Do other languages get this right 100% of the time on all platforms? I don't think so, it's just you've never noticed.
* C: has no concept of unicode strings per se, may or may not work depending on the implementation and how you choose to display them (CLI probably "works", GUI probably not)
* Rust: seems to assume UTF-8? (https://doc.rust-lang.org/std/ffi/index.html#conversions)
* Go: gets this right, but probably breaks on Windows? "string is the set of all strings of 8-bit bytes, conventionally but not necessarily representing UTF-8-encoded text" (https://golang.org/pkg/builtin/#string)
In short, either I don't understand what point you're making, or it isn't unique to Python.
https://tour.dlang.org/tour/en/basics/alias-strings
> This means that a plain string is defined as an array of 8-bit Unicode code units. All array operations can be used on strings, but they will work on a code unit level, and not a character level. At the same time, standard library algorithms will interpret strings as sequences of code points, and there is also an option to treat them as sequence of graphemes by explicit usage of std.uni.byGrapheme.
And perhaps my favorite part (https://tour.dlang.org/tour/en/gems/unicode):
> According to the spec, it is an error to store non-Unicode data in the D string types; expect your program to fail in different ways if your string is encoded improperly.
I should note that what I really like about this approach is the total lack of ambiguity. There is no question about what belongs in a string, and if it's not UTF then you had better be using a byte or ubyte array or you are doing it wrong by definition.
What I like about the D approach isn't that declaring it an error actually solves anything directly (obviously it doesn't) but that it removes any ambiguity about how things are expected to work. If the encoding of strings isn't well defined, then if you're writing a library which encodings are the users going to expect you to accept? Or worse, which encodings does that minimally documented library function you're about to call accept?
They’re conversions because they’re not UTF-8 in the first place, that is, they’re not String/str. The conversions are as cheap as we can make them. That language is meant to talk about converting from OsString to String, not from the OS to OsString.
This has led to developers attempting to conflate the two distinctly different concepts and make APIs support both while behaving differently.
A simple solution is there in plain sight: just use exclusively b"" and u"" strings for any code you wish to work in both Py2 and Py3, and forget about "". All and any libraries should be using those exclusively if they support both. Python3-only code should be using b"" and "" instead.
One could consider this a design oversight in Python 3: the fact that the syntax is so similar elsewhere makes people want to run the same code in both, yet a core type is basically incompatible.
from __future__ import unicode_literals
This has not been the case since 2012. The last release of Python 3 for which this was the case reached end of life in February 2016. Please stop misinforming people.
It's absolutely not correct, because there are many APIs which take "native strings" across versions aka they take an `str` (byte string) in Python 2 and an `str` (unicode string) in Python 3. unicode_literals causes significantly more problems than they solve.
The correct cross-platform solution (and the very reason why the `u` prefix was reintroduced after having initially been removed from Python 3) is in fact to use b"" for known byte strings, u"" for known text, and "" for "native strings" to interact with APIs which specifically need that.
I've not really had many other issues porting from 2 -> 3 from my own code; issues usually arise from 3rd-party libs that are relied upon (especially if they utilize C/C++ extensions). DB libs have sometimes been problematic. IIRC, pysybase requires you to set the encoding of strings now (which wasn't required before). I use pysybase to talk to both Sybase & MS SQL (it talks the TDS protocol).
???
The Python 3 CSV module works on text only (unicode), and you seem to be misreading the note: the module does in fact do newline transparency: https://docs.python.org/3/library/csv.html#id3
`newline=''` is to be specified on the file so it doesn't perform newline translation, because that would change the semantics of newlines in quoted strings: by default, `open` will universally translate newlines to `\n`, `newline=''` means it still does universal newlines (for purpose of iteration) but it doesn't do translation, returning newline sequences unchanged.
The Python 2 CSV module only worked on byte strings, and magic bytes.
It's better in some ways. But then, I gave up long ago on manipulating files with Python, because Python 3 simply decided that anything on my filesystem is utf-8 by default.
Want to get an stream out of a file? Better have utf-8 content. Otherwise all the documentation will be buried 15 meters deep, outdated on some aspects and dependent on some not-released yet features.
Want to name a file? Better have an utf-8 name. I gave-up on python before I was able to open a file with a name that doesn't conform to utf-8.
Want to read a line at a time? Sorry, we only do that to text. Go get an utf-8 file.
Want to match with a regular expression or parse it somehow? Sorry, we only do that to text.
And so on.
> Want to get an stream out of a file? Better have utf-8 content. Otherwise all the documentation will be buried 15 meters deep, outdated on some aspects and dependent on some not-released yet features.
f = open("myfile.txt", "r", encoding=ENCODING)
export NAME="$(head -c 5 /dev/urandom)"; touch "$NAME"; python -c 'import os; open(os.environ["NAME"])'
fh = open("foo", "rb"); lines=[l for l in fh]
import re; re.match(b"abc", b"abcdef")I lean towards "programmer problem" rather than "language problem".
Are they? Is the separator 0x2f or / ? Or are you talking about the filename only, not paths?
Similarly, Unicode normalization means that you can have multiple visually indistinguishable values which may or may not point to the same file depending on the OS and filesystem. I’ve had to help very confused users who were told that a file they could see wasn’t present because their tools didn’t handle that, too.
It's 0xf2. Encode a path to UTF-16 and watch the entire thing burn, with the FS seeing the path you provided end up right before or after the first / (because it encodes as either 0x00f2 or 0xf200 depending on BE versus LE).
(As other commenters have pointed out, Unix paths do not require a specific encoding, so robust applications cannot rely on any assumptions about encoding of existing files. But when creating new files, they must not try to encode paths as UTF-16.)
That's the point. The separator is not "/", because "/" would be a character to encode. The separator is a specific byte, and so is the terminator.
> Valid Unicode paths are encoded as UTF-8 on unix systems.
There is no such thing as "unicode paths" on UNIX systems, valid or invalid.
Yes, and you should treat them as such. If you decode them you're doing it wrong. os.listdir lets you treat paths like bytes and responds in kind:
https://docs.python.org/3.5/library/os.html#os.listdir
The problems you mentioned do exist, and Python 3's response is to make you deal with them. This is the whole point, not a mark against it. Often when you find that Python 3 is making it hard for you, it's because you're doing it wrong.
>You shouldn't decode them except if you need to for display (and in that case, you should be prepared to handle filenames which cannot be decoded as text).
If I create a file whose name consists entirely of bytes that correspond to non-printable characters, how would you display that in the language of your choice?
If I create a file whose name cannot possibly be valid UTF-8, how would you display that in the language of your choice?
Both of these things are legal to do on at least some systems. Presumably you have in mind a language which will magically handle this case with no extra effort from the programmer, though, so I'm curious to know the language and what it would do.
Of course you should. But the debate here is not about how a programmer should do it. The debate here is that Python3 doesn't do it that way by default and at the very least it takes extra code to do it, and as the article points out in some cases like command line arguments makes it impossible to do it that way.
A well designed computer language encourages good code by making it easy and obvious to write it. By that metric Python3's handling of bytes and strings is not well designed.
By the by I have been bitten by this. As others have mentioned a backup program turns out to be a worst case. I had to redo the string / file name handling twice before I was confident it was correct. I'm an experienced Python programmer both Python2 & 3. The final design wasn't obvious to me, and took substantially more code than I thought it would.
os.fsencode() was changed in Python 3 longer after the 3.0 fork and isn't inherent to 3 -- the same change could be made to Python 2. It only wasn't because Python.org has been intentionally neglecting Python 2 since 2015.
(The Python 2 equivalent of os.fsencode() is just encoding with sys.getfilesystemencoding(). The primary difference is that fsencode() uses the surrogateescape error handler by default. Since this is a relaxed behavior, it seems like it could be added to Python 2 without regressing existing programs.)
Which is a lossy process. The example given in the article is dealing with legacy filenames -- strings from environments where the source encoding isn't necessarily even known.
You're doing the classic Programmer Thing here where you assume that the real problem is choosing a correct algorithm. It's not. Real problems are about dealing with data, and you don't always have access to a time machine to fix the algorithms that created that data in the first place.
Honestly the poster is correct as far as it goes: modern python (and modern string libraries more generally) are pretty bad at handling dirty input. It's just not a problem area they've been interested in treating, because they'd like to work on stuff that feels like the correct algorithm instead of making tools for dirty data.
* OS treats filenames as bytes and allows arbitrary byte strings,
* Users want to edit filenames as text
are in conflict and you can't solve it without forcing some encoding or giving them a byte editor. The solution "filenames are bytes but we'll treat it as ASCII because 0x32-0x7E should be enough for anybody" is not acceptable for möst of thé wõrld.
This is incorrect: Python is fine for working with messy data. The difference is that when you CONVERT that data you have to handle the possibility of errors rather than hoping for the best. If you’re working with filenames, you can pass bytes around all day long without actually decoding them; it’s only when you decode them that you’re forced to handle errors.
Heh, "convert" it by concatenating it with a different directory path? Print it for the user? Stuff it into some kind of output format? Every one of those actions is going to toss an exception in python[1], and there are no tools available for reliably doing this in a safe and lossless way that matches the obvious syntax you'd use with "good" strings.
Maybe that's "fine" to you, I guess.
[1] And in lots of other environments. The quibble I have with the linked article is, again, that this kind of thinking is pervasive in modern string handling. Python certainly isn't alone.
Similarly once you’re talking about output other than passing it through unmodified to a format which can handle that, you are by definition converting it and need to handle what can go wrong. It’s easy to handle this in several ways, either by reporting an error or using some sort of escaped representation with an indication that this doesn’t match the document encoding, but you no long have the luxury of pretending that treating string as a synonym for bytes was ever safe.
I'm not a fan of Python3's language fork in any way (I think it was completely unnecessary to fork the language to make the improvements they wanted to) but I'll admit things like Pathlib and UTF-8b are a step in the right direction for handling arbitrary Unix paths. (I work with a large Python 2 code base and a product that has to interact with a variety of path encodings, so this subject is... sensitive for me.)
People don't think about these things because Python encourages you to ignore them. Until it doesn't. There'd be a lot less confusion if Python 3 were more strict/explicit about conversions.
Equality testing is implicitly done in a lot of places (like searching), it would become a bit of a hassle if the basic types were to throw exceptions when compared with each other.
>>> 1 == '1'
FalseFor something like Unicode, I think the point is that it "just worked" before where it would have been a lot more time and effort to get it "working" in say Java. Types and compile/runtime errors are a balance in the end that depends on the use case and if the programmer wants something that works (for the short term usually) or is resilient and will work long term. IMO, the influx in popularity of untyped languages (Javascript being used on the server side, ew) is a sign that so much coding today is not done for the long haul. I think there are good balances to be hit though, and Unicode is a place that highlights that python could use some better optional typing aid, as Typescript brought to JS.
For me it’s literally personal.
So your inability to handle basic stuff like this is actually costing you money.
"część" == "część".encode('<some-obscure-encoding>')
My point is, `s == b` is not a well-defined question. It only makes sense to ask it for a particular encoding: "Are these two things isomorphic according to cp-1252?"¹². Giving a type error is a decent way of signalling all this.
---
¹ And even then, I don't like automatic type conversions that enable things like `3 == "3"`, but I guess that's a matter of taste.
² Of course you could pick a convention like `s == b <=> s = b.decode("utf8")`, but that's a different question.
> I don't like automatic type conversions that enable things like `3 == "3"`
I didn't mention this, and Python, the language under discussion, doesn't do this.
> "Are these two things equivalent?" has an unambiguous answer regardless of whether the things are of different types.
as
"Isomorphic things should compare equal"
So, to respond to (what I think is) your actual point: in my view, equality is defined on values of a single type, so I prefer to distinguish
"a isn't equal to b" (= False)
and
"there's no defined way to compare a and b, because they have different types" (= some kind of error or a null-like value)
To me, conflating the two seems counterproductive, but perhaps this is personal preference (probably correlated with how much one likes static typing).
(I guess I ended up conflating equality and equivalence[1], but in my defense, most languages seem to do that too; and the presence of the `is` operator in Python mixes things up even more)
[1] https://en.m.wikipedia.org/wiki/Equality_(mathematics)#Relat...
Also, `==` doesn’t always answer the question of whether two things are equivalent in Python:
>>> 1.0 == 1
True
>>> {'a': 5} == OrderedDict({'a': 5})
True
>>> 0 > None
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '>' not supported between instances of 'int' and 'NoneType'Considering the expression "x > y", the full process is:
1. Call x.__gt__(y).
2. If that returns a bool, that's the result. If it raises an exception, propagate the exception.
3. If it returns the sentinel value NotImplemented, attempt the reflected version of the operation: y.__lt__(x).
4. If that returns a bool, that's the result. If it raises an exception, propagate the exception.
5. If it returns the sentinel value NotImplemented, raise TypeError and inform the programmer this operator is not valid on these types.
>>> a = OrderedDict([(1, 2), (3, 4)])
>>> b = {1: 2, 3: 4}
>>> c = OrderedDict(reversed(a.items()))
>>> a == b == c
True
>>> a == c
False> Also, `==` doesn’t always answer the question of whether two things are equivalent
because the examples you posteda kinda make it look like it does, i.e. `1.0 == 1` even though one is a float and the other is an int.
Absolutely not; that's basically the OG subjective question whose lack of an obvious right answer underpins a substantial quantity of the bugs in less strongly-typed languages.
Nor is an unambiguous answer possible symbolically, or in common logic, or the semantics of spoken languages.
That’s one of the reasons I love Swift and appreciate that it wants to get these things right, even if it means changing every year in the first few versions.
Only his third bit of advice isn’t wrong and treating it as something unusual shows the problem: the only safe way to handle text has always been to decode bytes as soon as you get them, work with Unicode, and then encode it when you send them out. Anything else is extremely hard to get right, even if many English-native programmers were used to being able to delay learning why for long periods of time.