I recently took care of a 2to3 migration involving a popular web framework and a few technologies for one of my employer's clients. The codebase is large and convoluted but I still found it surprisingly easier and way more interesting than I expected.
There is some actual difficulty in dealing with packages that have deprecated for years, and other gory details. But these days 2to3 drama is mostly a meme. The Python community, with a few exceptions, has done an amazing job at keeping things working.
If you want have a bug you want attention on in an upstream project, the easiest way to get it is to do a task for them first, and doing python 2 to 3 is a usually a pretty straightforward one.
> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.
Saying that "probably do some things different" is quite an understatement.
Python 2 to 3 was an articulated train collision in slow motion that spans a decade. I'm sorry to say that, but it was monumentally awful.
I really, really wish that it didn't happen because I think it kept Python back at least 5 years.
People got really burnt by this upgrade process, and the answer was "we said we'd do backwards incompatible changes". "And what, are we going to have this every few years?" one might have though.
So, it isn't enough to tell people to "stop complaining about Python 2to3" and things might be done differently in the future or past.
Yeah, it's done alright for itself hasn't it? Look at the stats, it's a top #2 language on Github, StackOverflow questions, Developer surveys or any other measure. Complaining that it could have achieved this 5 years earlier, I don't see the point. We can just appreciate where it's at now, and the people who were instrumental in getting us here.
Isn't this the current situation now? Granted, not as big as Python 2 to 3, but there are backwards incompatible changes often.
Breaking stuff as they did made no sense. And the problem is that after 10+ years that python3 came out in most Linux distribution if you type python you get python2, and it will probably ever will for maintaining compatibility with scripts that started with `#!/usr/bin/python` expecting it to be python2 and not 3. That is a problem for everyone, for maintainers of the distributions, for developer and sysadmins that cannot assume that the default version of python is 3 (you don't know how many time I wated debugging programs written by people that use Arch or whatever fancy distribution and assumed that /usr/bin/python is python3, and then the script breaks on the Debian production server...)
At least since they made a new language, they could invented a new extension for the files, so the `/usr/bin/python` executable could have detected the version to use based on the extension (launch .py3 files with python3, and .py files with python2).
Then they could have had a clear upgrade guide that is to run 2 to 3, then clear explanations on how to fix all the remaining string issues. Instead, I have no clue what 2 to 3 fixes and what it doesn't, leading to "2 to 3 will probably work, but if it doesn't, then you'll just have to figure it out yourself."
That's unacceptable, and as a result caused many libraries to take a long time to upgrade. Personally, I had to spend 10 YEARS to wait to upgrade to Python 3 due to the wxPython library that all of my projects used that took that long to upgrade to Python 3.
* If you do X you'll fail
* If you don't do X you'll fail
* X is neat but don't worry about it til later.
I imagine that the people in charge of python, a wildly popular programming language (at the time, and still), got all sorts of input, much of which conflicted. They chose a path and it was a bad choice, a mistake. This is clear in retrospect 13 years later, but at pycon in 2008 (a few months before python 3 officially released) the viewpoint was more mixed - a lot of people thought it was a good path and a lot of people thought it was terrible and others still thought the transition plan was annoying but would be cleared up pretty quick.
What drives me to ire to no end are the many people that defend it on message boards with “Just upgrade; you had ten years.”, clearly not realizing the risks and money involved in such an endeavor.
It's as though the government suddenly start putting something in the water that improves it's health benefits, but will erode contemporary plumping pipes away in ten years and people say. “Just break down your walls and replace your plumbing; you have ten years.”. — it is quite expensive, and risks that my house fall over.
I get why some people think a default integer division is a good idea. I do. Having said that, it was clear that Python was aiming itself at being a kind of glue language wherein various functions were obvious, and integer division is not obvious to someone picking up programming for the first time.
If I walk up to someone and ask "What's three divided by two?" I would expect more "One and a half" answers than "Two goes into three just the one time."
In Python 3, it always gives a floating-point result. You can use a floor division operator if you want to round everything (including integers) towards negative infinity.
See https://www.python.org/dev/peps/pep-0238/ for details.
> 20 / 15 = 1
> 20.0 / 15 = 1.3333333333333333
> 20 / 15. = 1.3333333333333333
> 20. / 15. = 1.3333333333333333
In Python 3:
> X / Y is ALWAYS float division
> X // Y is ALWAYS integer division
This was managable in Python 2 but led to lots of float(argument) calls in every function that wanted to do float division. That lead to unnecessary calls, and unnecessary noise in the code.
In Python 2: 20 / 15 = 1
In Python 3: 20 / 15 = 1.3333333333333333
To get the old behavior a new operator was added "//"
In all fairness, the C/C++ and Java communities were and are quite substantial.
Of course the rest of us ought to expect 4/3 as the result of 20/15 ;-)
Since Python 2 does not have any form of type annotation, this made the task hard to do automatically and error-prone.
I don't know what the PSF should have done to placate people. Keeping a core feature like division confusing like that isn't a good option, and extending support even longer seems ludicrous to me.
You are describing it as if it is for everyone easy to convert.
That might be true for fast-paced start-ups and web companies which rewrite their code every few years.
I work in science, in projects which have complex infrastructure and are developed in the course of many years. There is a vast amount of code which is written in the language and virtually nobody has time to rewrite and convert it. Me neither. As a consequence, Python is not really ideal for this kind of work, and I will tend to avoid it in my next job.
It might not seem to matter to you, but a good part of Python's success is actually because of things like Numpy, Scientific Python, and statistics and science libaries which were written by scientists. Some, like Konrad Hinsen (one of the authors of Numpy) are now even considering to move to another language:
http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stabilit...
The idea that converting is easy is also not generally true for any kind of infrastructure. Consider the Python Wiki:
At the bottom of the page, you find the note "MoinMoin powered". The Wiki software is the MoinMoin wiki software. It is the only large wiki software with multi-markup support written in Python. It has so far no Python 3 port. This is because people have written it in their spare time, and they have no time to rewrite it. Do you have the time? Would you volunteer for that?
From the point of a naïve language author or a library author, change is part of life and users of their software should just upgrade to the latest version, as you write. But this is not friendly to the users of the software. Users really have other things to do than to constantly rewrite their software according to your latest syntax change or broken dependency.
More proficient languages and infrastructure projects, like C, C++, Common Lisp, Java, and the Linux Kernel, do not have this approach - they maintain strict backwards compatibility. "WE DO NOT BREAK USER SPACE", in all caps - do you recognize that phrase? You can google for it. And this rule has a reason. You can think a few minutes how successful that project would have been without this rule. What if if you want to run a new Python on Linux, you first need to upgrade your Linux kernel, and then you need to upgrade your Linux distribution, and then you find that the new distribution release does not support HTML1, so the library you were going to use is broken, and, oh, your printer won't work with it? This is more or less what Python expects from its users.
The Python developers do not adhere to this - they break backwards compatibility. If one reads the fine print about the release cycle, it says clearly that there might be breaking changes even in minor releases.
At the same time, the Python developers did not want people to realize that Python 3 is not the same language, and used the same file extension for the new language. This has as an effect more backwards incompatibility, for example there are libraries which continue to support Python 2 for some time and then stop that support - without using a major version number increment. As a consequence, Python 2 projects which use such a library will break if using normal tools, which assume that semantic versioning is adhered to. In other words, one cannot rely on that Python library authors adhere to semantic versioning.
All this is, regardless how much Python developers fuss about it, a choice. It would have been possible to keep Python 3 backwards compatible without breaking anything. Common Lisp has done it and it is decades older. But this has not happened and as a consequence, Python is less suitable for some types of projects and infrastructure.
That's not something to complain about, and I don't. It is just a fact.
Have you done such an upgrade with a million line codebase?
No company wants to do that with such a large codebase with the potential of regressions that can cost quite a bit of money.
Reading this article makes it sound as though the experience taught him a valuable lesson and that he agrees that in retrospect Python 3 was a mistake, and it was.
There is quite a good reason why so many platforms and languages take indefinite backwards compatibility quit seriously. — it is simply a very demanding thing of large codebases to rewrite to a newer version of the language with all the potential of subtle regressions.
> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.
The article suggests that if they could go back in time, they would not do it and make no backwards incompatible change to Python 2.
> But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.
It was certainly not malicious, but it was indeed a bad idea which is what the complaining is: that Python 3, or any such backwards incompatible changes should have never happened.
This is why Rust uses editions and many other languages a system where backwards incompatible changes must explicitly be imported and the old behavior continues to be supported indefinitely.
Yes. With a transition plan everything was ratcheted over to python3 syntax, then test compatibility. Now developers no longer have to remember the nuance between python 2 and 3 and how to write compatibility layers for both.
Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.
Then tell me, what were the costs of the transition when completed? and what where the projected costs when starting?
> Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.
You wouldn't if Python 3 didn't exist either.
> You wouldn't if Python 3 didn't exist either.
This is a bit of a loony argument. There are so many useful features post py27 that may not have been possible without the breaking changes 2to3 introduced. Especially the horrific bytes/str situation in 2.7.
So your argument is that Py3 shouldn't have happened, despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?
This situation could easily have been solved without breaking changes in a multitude of ways.
> So your argument is that Py3 shouldn't have happened
Yes, my argument is absolutely that none of the changes warranted breaking backwards compatibility.
> despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?
That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.
> That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.
You haven't made an argument, nor given any alternatives. You've just complained that the free work of others wasn't to your specifications.
How would you propose handling the str/bytes situation without breaking backwards compatibility?
If it helps, you can convince yourself that Py3 is a different language that just happens to be very similar to Py2, and that Py2 needs new maintainers. This happens all the time.
Please, do elaborate. (I'll note that if I were redesigning python3, I wouldn't settle on the design for unicode that py3 uses, something more rust-like is probably a bit better, but there's no way to transition from string-literals-are-bytes to string-literals-are-unicode without breaking backwards compatibility)
It does exist. Python 2 is dead. The transition was more painful than it perhaps needed to be, but is IMO a better language for it than 2.7, or 2.6.
PEP-3000 was 15 years ago.
Complaining for decades about mistakes, that they admit, with hindsight, won't change anything.
You need to read it again. What he actually said was he should have provided more support to developers migrating code, not that he wouldn’t have done it.
Maintaining Python 2 indefinitely is an option actively being taken away from people. More importantly, fuck anyone who wants to direct my budget! There's things written in Python 2 ages ago that don't justify additional developer time in my company. Except the community is making us do it.
If you think migrating some ancient Django app to modern Django is trivially easy, why don't you come do it for us?
Where do you see this happening? All I see are people and organizations deciding that they don't want to do the work of maintaining Python 2.
It's not a tragedy, backwards-compatibility is a burden, and I understand maintainers, especially volunteers not wanting to do it. But migration is a pain, I found the conversion was not too hard, but testing the migrated scripts for breakage was time consuming.
FYI, the cause of my latest migration was that the latest version for the mysql-connector for python no longer works with 2.7. See the changelog entry for v8.0.24 here: https://github.com/mysql/mysql-connector-python/blob/master/...
So something to consider if you want to stay on Python 2 forever.
Python 2.7 is not getting any security/crypto updates backported. Eventually PyPI's SSL configuration will change to a point where Python 2.7 won't handle it and then that's it. Game over, no more pip for you, even if you froze on < 21.0.
There are Linux distros out there in LTS until late into this decade that shipped with Python 2.7. Ubuntu 18 comes to mind, which EOLs in 2028. They've got a real hard problem to solve.
You can maintain Python 2 yourself just fine, though, if you like.
There are several projects out there that are supporting python 2.7 with security updates. Tauthon is probably the biggest one.
It appears to only take away from people the option to use the unmaintained Python 2.
Btw you can still return your copy of Python for a full refund.
Just because Python core can say that they support for five years and act like they're being generous when they do so does not make them less like asshats. Most software is not a living, breathing thing.
Is your budget going at all to maintaining Python 2 or the ancient Django you use? I'm gonna guess not.
We all know about how COBOL is alive and kicking but Python's core team decided that reality doesn't apply to them.
Python is a fantastically productive language in many contexts. The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again. But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.