> But the incompatibility between Python 2 and Python 3 is perhap s only a symptom of a larger problem. The Python developers have decided that backwards compatibility is not that important any more.

Exactly: and that was a wrong decision for anybody but the developers of Python.

Everybody else prefers having something that works: "The improvements are welcome, but please allow us to to run our old programs too, thank you, and allow us to use that new feature only once we need it."

It's an obvious expectation. We would also hate a new version of a word processor which wouldn't open our old documents. Or a new version of Photoshop which wouldn't open our old pictures. Or a new version of the browser where only the newest web pages are visible.

It follows that it was absolutely technically possible to have a new version of Python in which the old programs still work. It's the failure of the developers that they haven't made it.

Compare that decision of theirs with the policy of Linus Torvalds who insists that the old user programs should never break on newer kernels.

As an example for Linus' "kernel changes should not break user programs" policy, he was famous for using harsh words to pass the message to those who tired to steer otherwise (these words I don't have to repeat, so I'll just quote his main message):

https://lkml.org/lkml/2012/12/23/75

"How long have you been a maintainer? And you still haven't learnt the first rule of kernel maintenance?

If a change results in user programs breaking, it's a bug in the kernel. We never EVER blame the user programs."

To explain that a bit, a Linux kernel maintainer is somebody who organizes and collects contributions of others, screens them, and when they are finished, passes them on to Linus to integrate them. That's a huge responsibility, and Linus uses such strong expressions only for people who have such responsibility and which do things which would harm his project. It is not the case that he talks like that to normal contributors (which should be guided by the maintainers).

And the reason why I think there is a deeper problem is that the issue is clearly not solved with the python2/python3 transition:

Why do some python developers have to maintain installations of a whole handful of python versions just to ensure that their code is working? Why all the mess with pyenv, virtualenv, and so on? If the python developers, as well as the library developers would support backwards compatibility, this would not be necessary at all.

> the issue is clearly not solved with the python2/python3 transition

Exactly. The incompatibility mess continues when using different 3.x versions.

Obligatory Reference to Rich Hickeys talk on versioning and interface specs:

https://www.youtube.com/watch?v=oyLBGkS5ICk

Discussion:

https://news.ycombinator.com/item?id=13085952

> It is, however, a huge problem for domains like scientific computing, where most code has no maintainers and even for very important code there is no budget or staff for maintenance

I think that's a tool selection problem, not just confined to the python world. If the language and libraries won't have a supported lifespan that matches with the maintenance budget of the projects using them then the wrong tool was chosen. If a project is expected to have a 10+ year lifespan of little to no maintenance then it needs to be built on languages/libraries that will have supported versions for that long.

Well, first thing is that most new code in scientific research is developed in PhD projects which last some three or maybe four years. The people who develop this do not have resources and time to maintain this code. Projects don't have a budget for that. There are projects which are very long-running (think CERN or ESO's VLT) but even there the true duration of the code usage is seldomly planned (AFAIK, ESO VLT has just started to transition from Tk/Tcl to Python).

You could say then, "well, then Python is perhaps just not a good match for those pesky scientists".

And this brings up two more points:

* A lot of important tools and libraries in the Python ecosystem was developed by scientists. Numarray/Numpy is a good example.

* If the core Python developers don't have the intention to maintain a backwards-compatible language version for more than, say, 15 years, they should perhaps clearly state on the python.org main page something like: "great, as you are a scientist, we welcome your contribution, but Python might not be suitable for tools that support long-term research".

"ESO VLT has just started to transition from Tk/Tcl to Python" - they might regret this, new Tcl releases tend to take backward compatibility much more seriously than Python :-)

Yes, I've always thought that Tcl should be promoted as an archival quality programming language.

> It is, however, a huge problem for domains like scientific computing, where most code has no maintainers and even for very important code there is no budget or staff for maintenance:

Data science is doing just fine, in fact is leading the migration: https://www.jetbrains.com/research/python-developers-survey-...

"data science" and "science" are quite different things. Science is the systematic and collaborative pursuit of knowledge in a long-term endeavour. It is based on sharing and open exchange of methods and tools. Numericas, mathematics and computational codes are just important tools to do that. As Hinsen in the blog post I cited above points out, the most part of important computational codes is written in one-off research projects which go for a few years, and the people who develop these codes normally have to move on and work for a different institution, if they manage to keep working in science. On the other hand, important codes and algorithms may be used for many many years.

"Data science" is a broad term but usually just means the application of numeric, and sometimes scientific, tools to commercial means. It is almost always done in companies. Typically, between such companies there is no open exchange of tools and methods, no exchange of knowledge, and no long-term use of generated codes. This is the reason why data science companies don't have the problems which Hinsen pointed out. But, they could become affected by a degrading suitability of Python for computational science, because their tools were initially developed by scientists.

Discussion of Hinsen's post on HN:

https://news.ycombinator.com/item?id=17058269