News for hackers. Stuff that matters.
Re:Mercurial’s journey to and reflections on Python 3 by jharsman on Monday January 13, 2020 @04:56PM (#22039155)
I was an early adopter of Mercurial and the teams insistence that file names were byte strings was the cause of lots of bugs when it came to Unicode support.

For example, when I converted our existing Subversion repository to Mercurial I had to rename a couple of files that had non ASCII characters in their names because Mercurial couldn't handle it. At least on Windows file names would either be broken in Explorer or in the command line.

In fact I just checked and it is STILL broken in Mercurial 4.8.2 which I happened to have installed on my work laptop with Windows. Any file with non ASCII characters in the name is shown as garbled in the command line interface on Windows.

I remember some mailing list post way back when where mpm said that it was very important that hg was 8-bit clean since a Makefile might contain some random string of bytes that indicated a file and for that Makefile to work the file in question had to have the exact same string of bytes for a name. Of course, if file names are just strings of bytes instead of text, you can't display them, or send them over the internet to a machine with another file name encoding or do hardly anything useful with them. So basic functionality still seems to be broken to support unix systems with non-ascii filenames that aren't in UTF-8.

Re:Mercurial’s journey to and reflections on Python 3 by pdonis on Monday January 13, 2020 @05:54PM (#22039660)
> the teams insistence that file names were byte strings was the cause of lots of bugs when it came to Unicode support

File names are a different problem because Windows and Unix treat them differently: Unix treats them as bytes and Windows treats them as Unicode. So there is no single data model that will work for any language.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @02:01AM (#22042509)
The Rust standard library has a solution for this that actually works: On Unix-like systems file paths are sequences of bytes and most of the time the bytes are UTF-8. On Windows, they are WTF-8, so the API users sees a sequence of bytes and most of the time they match UTF-8.

This means that there's more overhead on Windows, but it's much better to normalize what the application programmer sees across POSIX and NT while still roundtripping all paths for both than to make the code unit size difference the application programmer's problem like the C++ file system API does.

Re:Mercurial’s journey to and reflections on Python 3 by pdonis on Tuesday January 14, 2020 @05:25PM (#22049872)
> On Windows, they are WTF-8

Seems like an apt acronym for Windows... :-)

On a more serious note, Python seems to have done something fairly similar with the pathlib standard library module.

Re:Mercurial’s journey to and reflections on Python 3 by simias on Monday January 13, 2020 @06:06PM (#22039761)
Not to mention case-sensitivity issues. Can you have two files, one named "FILE.txt" and the other "file.txt" in the same directory for instance?
Re:Mercurial’s journey to and reflections on Python 3 by SSLy on Monday January 13, 2020 @06:40PM (#22040043)
On windows? Of course you can.
Re:Mercurial’s journey to and reflections on Python 3 by edgyquant on Monday January 13, 2020 @07:53PM (#22040639)
I'm certain you can on Linux as well. Only Macs old HFS would not allow it.
Re:Mercurial’s journey to and reflections on Python 3 by cataflam on Monday January 13, 2020 @08:10PM (#22040741)
Isn't this a fairly recent change?
Re:Mercurial’s journey to and reflections on Python 3 by amaranth on Tuesday January 14, 2020 @12:18AM (#22042070)
NTFS has always been case sensitive, Windows API just lets you treat it as case insensitive. If you pass `FILE_FLAG_POSIX_SEMANTICS` to `CreateFile` you can make files that differ only in case.
Re:Mercurial’s journey to and reflections on Python 3 by mathw on Tuesday January 14, 2020 @03:56AM (#22043009)
Good luck using those in some tools which use the API differently though. Windows filenames are endless fun. What's the maximum length of the absolute path of a file? Why, that depends on which API you're using to access it!
Re:Mercurial’s journey to and reflections on Python 3 by rurban on Wednesday January 15, 2020 @12:39AM (#22052001)
Even worse on Unix where it depends on the mount type. Haven't seen much proper long filename support in Unix apps or libs, it's much better in Windows land. Garbage in garbage out is also a security nightmare as names are not identifiable anymore. You can easily spoof such names.
Re:Mercurial’s journey to and reflections on Python 3 by gpderetta on Monday January 13, 2020 @05:15PM (#22039315)
Hum, any program that doesn't treat filenames as bytestreams on unix is broken. Doubly so if its primary purpose is preserving and archiving files.

Are you sure the issue wasn't something else?

Re:Mercurial’s journey to and reflections on Python 3 by lmm on Monday January 13, 2020 @07:10PM (#22040279)
The point is that filenames aren't bytestreams on windows, and if you treat them as such then your program won't work.
Re:Mercurial’s journey to and reflections on Python 3 by WorldMaker on Monday January 13, 2020 @07:58PM (#22040670)
By this point, any cross-platform file tool that isn't using Unicode as a lowest-common denominator for filenames and similar things to insure maximal compatibility is likely ready to cause havoc.

(The remarks in the post here that Mercurial on Python 3 on Windows is not yet stable and showing a lot of issues is possibly even an indicator/canary here. To my understanding, Python 2 Windows used to paper over some of these lowest common denominator encoding compatibility issues with a lot more handholding than they do with the Python 3 Unicode assumption.)

Re:Mercurial’s journey to and reflections on Python 3 by lmm on Monday January 13, 2020 @09:12PM (#22041173)
> By this point, any cross-platform file tool that isn't using Unicode as a lowest-common denominator for filenames and similar things to insure maximal compatibility is likely ready to cause havoc.

Be that as it may, Mercurial has existing repositories that may use non-unicode filenames, and just crashing whenever you try to operate on them is probably not an acceptable way forward.

Re:Mercurial’s journey to and reflections on Python 3 by WorldMaker on Tuesday January 14, 2020 @02:39PM (#22048434)
Sure, but that's also not the only resulting option; instead of erroring you could also do something nice like help those users migrate to cleaner Unicode encodings of their filenames by asking them to correct mistakes or provide information about the original encoding. It takes more code to do that than just throwing an error, of course, but who knows how many users that might help that don't even realize why their repositories don't work correctly on, say, Windows.
Re:Mercurial’s journey to and reflections on Python 3 by Dylan16807 on Tuesday January 14, 2020 @03:09AM (#22042801)
Windows filenames basically are bytestreams. But the bytes come in pairs.
Re:Mercurial’s journey to and reflections on Python 3 by lmm on Wednesday January 15, 2020 @02:51AM (#22052426)
Not really. Certain byte sequences are invalid.
Re:Mercurial’s journey to and reflections on Python 3 by Dylan16807 on Wednesday January 15, 2020 @03:14AM (#22052514)
Certain byte sequences are invalid in unix filenames too. So that can't be the factor that decides if they are bytestreams or not.
Re:Mercurial’s journey to and reflections on Python 3 by xorcist on Monday January 13, 2020 @05:28PM (#22039447)
If hg borked on non-ascii characters, it sounds like the problem was rather that it didn't treat that data as a bag-of-bytes. Not the other way around?
Re:Mercurial’s journey to and reflections on Python 3 by ploxiln on Monday January 13, 2020 @05:52PM (#22039641)
He was trying to use Windows. For Windows, you pretty much have to go through unicode to utf-16, can't be arbitrary bytes, can't be utf8.

(I think that relatively recently it is possible to use utf8 with some new windows interfaces ... but this is probably not widely compatible with older windows releases ...)

Re:Mercurial’s journey to and reflections on Python 3 by Dylan16807 on Tuesday January 14, 2020 @03:06AM (#22042792)
Windows uses arbitrary shorts that are sort of supposed to be utf-16. Just like Unix uses arbitrary bytes that are sort of supposed to be utf-8.

You have to convert between them, but neither uses proper Unicode to represent filenames.

Re:Mercurial’s journey to and reflections on Python 3 by cbsmith on Monday January 13, 2020 @05:56PM (#22039672)
Yeah, but utf-16 is still bytes. It's just bytes with a different encoding.

But I do see the pain with Python 3 where the runtime tries to hide these kinds of issues from you. That abstraction can make it difficult to have the right behaviour.

Re:Mercurial’s journey to and reflections on Python 3 by mynegation on Monday January 13, 2020 @07:33PM (#22040469)
Everything is bytes but the meaning assigned to bytes, matters. Let’s say I create a file named «Файл» on Unix in UTF8 and put it into git repo. For Unix it is a sequence of bytes that is representation of Russian letters in UTF8. So far so good. Now I clone this repo to Windows, what should happen? The file can not be restored with the name as encoded into bytes on Unix, that will be garbage (that even has a special name “Mojibake”) in the best case or fail outright in the worst. What should happen is decoding of those bytes from UTF8 (to get original Unicode code points) Into Unicode code points, then encoding using Windows native encoding (UTF-16).
Re:Mercurial’s journey to and reflections on Python 3 by mikepurvis on Monday January 13, 2020 @08:43PM (#22041001)
True, but one of those representations still needs to be canonical one in the repo for the purposes of hashing into the commits and so on.

Git builds a bunch of logic like this in around handling line endings in text files.

Re:Mercurial’s journey to and reflections on Python 3 by cbsmith on Monday January 13, 2020 @08:02PM (#22040696)
Everything isn't bytes. Strings without an encoding don't have a specific byte representation.
Re:Mercurial’s journey to and reflections on Python 3 by takeda on Tuesday January 14, 2020 @12:48AM (#22042204)
It's the other way around. Strings always have meanings and always reference the same characters. You use encoding to encode strings into bytes.

Bytes without encoding, don't have any meaning, they are just... random bytes.

Re:Mercurial’s journey to and reflections on Python 3 by cbsmith on Wednesday January 15, 2020 @04:22PM (#22058772)
We're actually saying the same thing. You're saying without an encoding you can't turn bytes into a string (technically, in Python terminology, that's a decoding, but you know... ;-). I'm saying a string doesn't have a byte representation without an encoding. That's two perspectives on the same truth.

I absolutely agree that a string has meaning without a byte representation. That's the whole point of having it as a distinct type.

Re:Mercurial’s journey to and reflections on Python 3 by lmm on Monday January 13, 2020 @07:11PM (#22040294)
UTF-16 is not "just bytes". There are sequences of bytes that are not valid UTF-16, so if you want to roundtrip bytes through UTF-16 you have to do something smarter than just pretending the byte sequence is UTF-16.
Re:Mercurial’s journey to and reflections on Python 3 by cbsmith on Monday January 13, 2020 @08:01PM (#22040687)
Sorry, I wasn't trying to imply that any permutation of bytes would work. If you encode it improperly, it's not going to work.