Re:Ride the carousel of folly with a Python migration
by deathanatos on Monday January 23, 2017 @09:21PM
(#13468302)
> The only somewhat problematic code was this:
for line in iter( handle.stdout.readline, '' ):
self.output.put( ( handle, line ) )
> This no longer terminated an emitted lots of odd values (one number repeatedly). It turns out it returns an empty b'' now when empty, as opposed to a ''. This could have been avoided had this API sensibly returned None instead of an empty string.
For any observers, this is much more idiomatically written:
for line in handle.stdout:
self.output.put((handle, line))
That is, the default iteration over a file-like object is line by line. Frankly, I wonder how many Python programmers are aware that iter() has a two-arg form; this is the first time I've ever seen it used (and I had to look it up myself).
> [PyPI]
His solution of maintaining separate code bases is … unusual. I feel like a cursory glance around the library landscape shows that most of the projects out there are maintaining a single codebase that supports 2.7+. six greatly simplifies most of this. I'm not aware of any projects that are supporting Python 2 and 3 concurrently by separating out their code completely.
> I presumed that I’d just open a “Migration from Python 2 to 3 Guide” and follow the instructions. Alas, there is no such guide. There appears to be no central resource whatsoever on the porting of code from version 2 to version 3.
An I'm Feeling Lucky search on that exact quote pulls up the migration guide in Python's official documentation.[1]
Re:Ride the carousel of folly with a Python migration
by agf on Monday January 23, 2017 @09:28PM
(#13468327)
In addition to what you've pointed out, he claims that 2to3 didn't do anything to his code, and yet he had to fix print statements. 2to3 definitely fixes print statements; he likely didn't read the usage or docs at all. By default it prints a diff, you have to use 2to3 -w if you want it to modify files on disk -- I certainly don't want tools modifying files in-place by default, so that seems like the right API to me.
Re:Ride the carousel of folly with a Python migration
by dr_zoidberg on Tuesday January 24, 2017 @12:03AM
(#13468941)
> By default it prints a diff, you have to use 2to3 -w if you want it to modify files on disk -- I certainly don't want tools modifying files in-place by default, so that seems like the right API to me.
Well, running 2to3.py -w -n -o [new_py3_path] [old_py2_path] got me a copy of a project with every fix already applied. Another alternative I've been thinking is doing a "last before 2to3 commit" (and leaving no pending changes), and then run 2to3 inplace and check those from the IDE (which may or may not be easier to a few folks that work with me).
I have to check a few more things, but it seems 2017 is the year I'm finally porting then two biggest projects I work on to Python 3. Luckily both projects aren't heavy offenders and should still remain Py2 compatible, but we may have the push with the clients to ask for exlusively Py3 support.
Re:Ride the carousel of folly with a Python migration
by brianshaler on Tuesday January 24, 2017 @12:29AM
(#13469041)
You could also test the waters with a `2to3` branch.
If you have uncommitted changes, instead of dropping them in as "last before..." you can git-stash them before branching and git-stash-pop them when you come back.
Re:Ride the carousel of folly with a Python migration
by dr_zoidberg on Tuesday January 24, 2017 @11:23AM
(#13472677)
Part of wanting no uncommited changes is to push a few issues that have been lingering around as "too easy, not important" and get the team to fix them once for all. But yeah, we could be a bit more "correct to the git way".
Re:Ride the carousel of folly with a Python migration
by throwanem on Tuesday January 24, 2017 @12:14PM
(#13473281)
At which point you need to run 2to3 again, and maybe update callers of the code you stashed, depending on what it actually does. If you know with certainty it isn't going to be affected in any way by the upgrade tool, of course, then either works as well. But otherwise I'd argue strongly for finishing the changes first, committing them, and only then running the upgrader.
It's an argument of preference in practice, and therefore all but guaranteed to go nowhere. I tend to think that updating a whole codebase like this is best done atomically rather than piecemeal, especially when the code belongs to a team rather than an individual. But I can see where opinions might differ on such a point.
Re:Ride the carousel of folly with a Python migration
by mortoray on Tuesday January 24, 2017 @01:46AM
(#13469317)
The simple syntax changes I did prior to running 2to3 as they came up on the first run I tried. The semantic changes that didn't come up in 2to3 is what I was more bothered about.
Re:Ride the carousel of folly with a Python migration
by zzleeper on Monday January 23, 2017 @11:38PM
(#13468844)
About PyPI, it is a bit messy.
I backported a py3 project to use py2 (on behalf of a few users) and it was a real pain in the ass to have a wheel that allowed for both versions.
I ended up having a py3 build on github, and using a TravisCI build that ran pasteurize and pushed to PyPI that version. At this point, I don't even know what the travis.yml file is doing (100 lines), and regret a bit backporting the project.
Re:Ride the carousel of folly with a Python migration
by slavik81 on Tuesday January 24, 2017 @06:58AM
(#13470652)
Interesting. I do stick to Python 2 features, but put in some work to ensure my programs are valid Python 3. I'm not a very skilled Python developer, so the simplicity of that strategy is attractive.
One thing that I notice is that your project is a library, while mine is a program. I wonder if that also plays a role.
Re:Ride the carousel of folly with a Python migration
by mortoray on Tuesday January 24, 2017 @01:49AM
(#13469330)
The `iter` form of the `readline` doesn't load the entire stream first, whereas the second form you posted does. This makes a significant difference when reading data from another program.
I saw that migration guide, but it has very few details about what is actually involved in migration. It doesn't include API change details.
Re:Ride the carousel of folly with a Python migration
by agf on Tuesday January 24, 2017 @01:57AM
(#13469369)
Why do you think the `for` loop form loads the whole stream? That shouldn't be true, since `for` just calls `iter` on the object internally, and normal file-like objects by default are iterated line-by-line.
If you saw the migration guide but found it incomplete, you should edit your post to say that rather than saying there isn't one.
Re:Ride the carousel of folly with a Python migration
by pdw on Monday January 23, 2017 @08:51PM
(#13468163)
He claims there's no migration guide, but when I google on the phrase he used, "Migration from Python 2 to 3 Guide", or any variation of that phrase, the first hit is https://docs.python.org/3/howto/pyporting.html
I wonder why he didn't find it.
Re:Ride the carousel of folly with a Python migration
by mortoray on Tuesday January 24, 2017 @01:50AM
(#13469338)
That is not an actual migration guide. It's merely a meta-doc saying what types of things you need to pay attention to. I was looking for a specific list of things that would have to be migrated. The docs mention of reading the release notes for each versino of Python 3 isn't very helpful.
Re:Ride the carousel of folly with a Python migration
by zzleeper on Monday January 23, 2017 @11:34PM
(#13468826)
In my opinion/experience, there are some important hidden costs of staying in py2:
1) Hard-to-read or quirky code. Unicode handling, args, harder unpacking, etc. The more time it passes, the more updates py3 gets, and the more outdated the py2 code looks.
For instance, after a year almost exclusively in py3, I have completely forgotten about iter(), unicode gimmics, etc. and also rely more on stuff that is just too slow on py2 (like ordered dicts). With 3.6, I can start assuming dicts are ordered, which will create further breaks for py2 users (if they want to backport anything).
2) Outdated libraries. As the other comments say, epydoc is abandonware. Had he migrated to py3 earlier, he would have stopped investing in that tool earlier and switched to the standard (sphynx).
3) Harder to interact with others in the community.
Although the folks at google might think otherwise, most people are now using py3, and most code being developed right now is on py3 (at least open source code).
If you are still on py2, it will be harder to hire (it hints at huge technical debt, and a tedious working environment), harder to sell to others (e.g. I don't use google compute because of that), and harder to collaborate on (for open source projects).
All in all, the more that py2 and py3 diverge, the harder it will be for legacy (py2) users to migrate (this was not the case a few years ago, when there were still important libraries on py2 only).
Re:Ride the carousel of folly with a Python migration
by rtpg on Monday January 23, 2017 @11:40PM
(#13468853)
Nitpick, but you cannot assume dicts are ordered in Py3, even in 3.6. CPython's 3.6 implementation will make it so, but it's totally a CPython implementation detail. Please use ordered dicts when appropriate (they'll be dicts in CPython anyways)
Re:Ride the carousel of folly with a Python migration
by lowmagnet on Monday January 23, 2017 @10:00PM
(#13468464)
I ran a 2to3 last week on a codebase I've been working for the past six months. Took me one try to get that I was missing a flag. 'git diff' cured me of thinking it wrote anything, and I found '-w' shortly after.
I guess a masochist could do something like pipe 1 to 'patch -p0' or something…
I ran into the same unicode issues as the author, and dumped all the code that handled it because I didn't need it in 3.
I also reduced the whole codebase by some 150 lines because of not needing to do a bunch of extra work I originally designed in when working through the code. This was more due to fresh eyes.
ETA: the code was on 2.7 because a library I needed wasn't yet available under python 3 (FuelSDK from SalesForce) until November, and I completely missed this. 1.0 version supports py 3, but it wasn't in pip.
Re:Ride the carousel of folly with a Python migration
by pdonis on Monday January 23, 2017 @09:56PM
(#13468449)
This statement in the article...
"there is nothing in the PyPI package description that actually indicates what version of Python is being targeted"
is incorrect. There are Trove classifiers[1] for both versions, and indeed for every minor version number of both versions. There are even Python 2 and 3 classifiers that end in "Only", which look like they would exactly fit the author's use case.
Re:Ride the carousel of folly with a Python migration
by mortoray on Tuesday January 24, 2017 @01:51AM
(#13469342)
Okay, they are classified but I don't see that anything in the setup, or pip package management, seem to actually use them. Are they somehow used in actual package management, or just search qualifiers?
Re:Ride the carousel of folly with a Python migration
by di on Tuesday January 24, 2017 @06:35AM
(#13470562)
You are correct, the classifiers are just search qualifiers.
However, since pip 9.0.0, it will respect the Requires-Python[0] metadata, which can be specified via your setup.py file.
Since PyPI allows only a single source distribution per release, though, this will still not get you separate modules for separate codebases as it seems you originally desired. As mentioned in a previous comment, it's generally preferred to support multiple versions in a single codebase via six.
Re:Ride the carousel of folly with a Python migration
by bigmanwalter on Monday January 23, 2017 @09:02PM
(#13468230)
Python 2 > 3 is the next Y2K. My colleague and I have started offering Python 2 > 3 consulting services as some of our clients weren't prepared to make the migration themselves. With Django announcing that they will drop Python 2.7 support, this might become one of our more popular services!
Re:Ride the carousel of folly with a Python migration
by viraptor on Monday January 23, 2017 @09:16PM
(#13468281)
I'm glad someone actually wrote up their migration. Even with some weird elements ("missing" guide?) it still contains good links and may be useful for others. Hope to see more of those, as opposed to the usual FUD and ideological arguing about py3.
Re:Ride the carousel of folly with a Python migration
by Animats on Tuesday January 24, 2017 @01:50AM
(#13469336)
That doesn't sound so bad. When I ported a medium sized system to Python 3 two years ago, I found serious bugs in four library packages. It took me about a month to get everything working. He's not reporting anything like that.
For any observers, this is much more idiomatically written:
That is, the default iteration over a file-like object is line by line. Frankly, I wonder how many Python programmers are aware that iter() has a two-arg form; this is the first time I've ever seen it used (and I had to look it up myself).> [PyPI]
His solution of maintaining separate code bases is … unusual. I feel like a cursory glance around the library landscape shows that most of the projects out there are maintaining a single codebase that supports 2.7+. six greatly simplifies most of this. I'm not aware of any projects that are supporting Python 2 and 3 concurrently by separating out their code completely.
> I presumed that I’d just open a “Migration from Python 2 to 3 Guide” and follow the instructions. Alas, there is no such guide. There appears to be no central resource whatsoever on the porting of code from version 2 to version 3.
An I'm Feeling Lucky search on that exact quote pulls up the migration guide in Python's official documentation.[1]
[1]: https://docs.python.org/3/howto/pyporting.html
Heck. I haven't done any 2 to 3 porting. I just read a few articles out of idle curiosity once and even I know 2to3 doesn't work like that.
Well, running 2to3.py -w -n -o [new_py3_path] [old_py2_path] got me a copy of a project with every fix already applied. Another alternative I've been thinking is doing a "last before 2to3 commit" (and leaving no pending changes), and then run 2to3 inplace and check those from the IDE (which may or may not be easier to a few folks that work with me).
I have to check a few more things, but it seems 2017 is the year I'm finally porting then two biggest projects I work on to Python 3. Luckily both projects aren't heavy offenders and should still remain Py2 compatible, but we may have the push with the clients to ask for exlusively Py3 support.
If you have uncommitted changes, instead of dropping them in as "last before..." you can git-stash them before branching and git-stash-pop them when you come back.
It's an argument of preference in practice, and therefore all but guaranteed to go nowhere. I tend to think that updating a whole codebase like this is best done atomically rather than piecemeal, especially when the code belongs to a team rather than an individual. But I can see where opinions might differ on such a point.
I backported a py3 project to use py2 (on behalf of a few users) and it was a real pain in the ass to have a wheel that allowed for both versions.
I ended up having a py3 build on github, and using a TravisCI build that ran pasteurize and pushed to PyPI that version. At this point, I don't even know what the travis.yml file is doing (100 lines), and regret a bit backporting the project.
All I had to do was create a setup.cfg that said:
To build, I use: The .travis.yml is just 7 lines. https://github.com/cgmb/guardonce/blob/v2.0.0/.travis.ymlIf you want to do both, then you need pasteurize, and end up with a travis mess: https://github.com/sergiocorreia/panflute/blob/master/.travi...
To be honest, for me the morale of the story is that supporting py2 is just not worth it. I am not going to use it, so why support it at all?
One thing that I notice is that your project is a library, while mine is a program. I wonder if that also plays a role.
I saw that migration guide, but it has very few details about what is actually involved in migration. It doesn't include API change details.
If you saw the migration guide but found it incomplete, you should edit your post to say that rather than saying there isn't one.
Interesting article though, thank you for posting it.
I guess if everything went smoothly, he wouldn't have had a blog post to write.
1) Hard-to-read or quirky code. Unicode handling, args, harder unpacking, etc. The more time it passes, the more updates py3 gets, and the more outdated the py2 code looks.
For instance, after a year almost exclusively in py3, I have completely forgotten about iter(), unicode gimmics, etc. and also rely more on stuff that is just too slow on py2 (like ordered dicts). With 3.6, I can start assuming dicts are ordered, which will create further breaks for py2 users (if they want to backport anything).
2) Outdated libraries. As the other comments say, epydoc is abandonware. Had he migrated to py3 earlier, he would have stopped investing in that tool earlier and switched to the standard (sphynx).
3) Harder to interact with others in the community.
Although the folks at google might think otherwise, most people are now using py3, and most code being developed right now is on py3 (at least open source code).
If you are still on py2, it will be harder to hire (it hints at huge technical debt, and a tedious working environment), harder to sell to others (e.g. I don't use google compute because of that), and harder to collaborate on (for open source projects).
All in all, the more that py2 and py3 diverge, the harder it will be for legacy (py2) users to migrate (this was not the case a few years ago, when there were still important libraries on py2 only).
Citation? Still feels about 50/50 to me and even that number feels a little generous to Py3.
I guess a masochist could do something like pipe 1 to 'patch -p0' or something…
I ran into the same unicode issues as the author, and dumped all the code that handled it because I didn't need it in 3.
I also reduced the whole codebase by some 150 lines because of not needing to do a bunch of extra work I originally designed in when working through the code. This was more due to fresh eyes.
ETA: the code was on 2.7 because a library I needed wasn't yet available under python 3 (FuelSDK from SalesForce) until November, and I completely missed this. 1.0 version supports py 3, but it wasn't in pip.
"there is nothing in the PyPI package description that actually indicates what version of Python is being targeted"
is incorrect. There are Trove classifiers[1] for both versions, and indeed for every minor version number of both versions. There are even Python 2 and 3 classifiers that end in "Only", which look like they would exactly fit the author's use case.
[1] https://pypi.python.org/pypi?%3Aaction=list_classifiers
However, since pip 9.0.0, it will respect the Requires-Python[0] metadata, which can be specified via your setup.py file.
Since PyPI allows only a single source distribution per release, though, this will still not get you separate modules for separate codebases as it seems you originally desired. As mentioned in a previous comment, it's generally preferred to support multiple versions in a single codebase via six.
[0]: https://www.python.org/dev/peps/pep-0345/#requires-python
Sphinx is close to a standard nowadays.