News for hackers. Stuff that matters.
Re:Mercurial’s journey to and reflections on Python 3 by libria on Monday January 13, 2020 @01:52PM (#22037347)
> Why am [I] doing this?

Lack of security updates past 2019 forced our hand. Did you find a way around that?

Re:Mercurial’s journey to and reflections on Python 3 by falcolas on Monday January 13, 2020 @02:39PM (#22037797)
> Lack of security updates past 2019 forced our hand. Did you find a way around that?

Amazon is maintaining Python 2 for at least 4 years, as part of their Amazon Linux long term support release. Google app engine will support Python 2 for an unknown amount of time; they haven't announced an end date. PyPy is Python 2, with (to the best of my limited knowledge) no plans to deprecate support. There are also other LTS releases out there which include Python 2 support.

IOW, the forcing function of the PSF no longer supporting Python is not as big a factor as was hoped.

Re:Mercurial’s journey to and reflections on Python 3 by rst on Monday January 13, 2020 @04:32PM (#22038882)
Security updates in Python itself aren't the only issue; a Python 2 project may also depend on packages with security issues of their own, which require continued upstream maintenance.

For example, the python-saml package (for managing SAML-based single sign-on) has separate Python 2 and Python 3 versions, and implements a security-sensitive protocol which means it has (in the fairly recent past) gotten security updates for issues serious enough to rate an assigned CVE. If you're using it, having the current maintainers walk away from the Python 2 version is a serious risk...

Re:Mercurial’s journey to and reflections on Python 3 by jnwatson on Monday January 13, 2020 @05:07PM (#22039253)
I'm a maintainer for a somewhat popular Python package that had support all the way back to 2.4, but I've had to systematically remove support for those versions. The problem is all the CI infrastructure and testing packages are removing support.

Is Amazon planning to support pytest for at least 4 years? It will have its last 2.7-supporting release very soon.

Re:Mercurial’s journey to and reflections on Python 3 by viraptor on Monday January 13, 2020 @03:25PM (#22038212)
This would only help the server side of mercurial though. There's no client-side supported distribution really. Pypy is not that popular yet.
Re:Mercurial’s journey to and reflections on Python 3 by falcolas on Monday January 13, 2020 @03:51PM (#22038424)
I don't know about others, but when I used Mercurial, it was via installing it through brew. And if brew installed pypy as a dependency so Mercurial could still use Python 2, I probably wouldn't have noticed.
Re:Mercurial’s journey to and reflections on Python 3 by viraptor on Monday January 13, 2020 @08:58PM (#22041098)
You'd notice, because it wouldn't work: https://www.mercurial-scm.org/wiki/PyPyPlan
Re:Mercurial’s journey to and reflections on Python 3 by kick on Monday January 13, 2020 @02:51PM (#22037912)
PyPy is keeping Python 2 support indefinitely, I believe.
Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Monday January 13, 2020 @01:58PM (#22037415)
There's a project for keeping Python 2 alive: https://github.com/naftaliharris/tauthon

It's particularly uncool that Guido brought up the prospect of lawyers (https://github.com/naftaliharris/tauthon/issues/47#issuecomm...) to force it not to be called Python and opposed to letting people who care about keeping Python 2 alive evolve it as "Python 2". (I know he has the legal right to insist on the name change. Still uncool.)

Re:Mercurial’s journey to and reflections on Python 3 by simias on Monday January 13, 2020 @03:07PM (#22038067)
I understand your point of view but on the other hand we can make the parallel with Perl 5 and 6. Having incompatible forks of the language share the same name is a pain for everyone involved. I can completely understand the "mainline" python maintainers not wanting to have to deal with that.

Besides if the Tauthon people are serious about maintaining their fork long term it needs to become more than a mere fork and a real language ecosystem of its own, in the long run having a different name will probably help with that, assuming that they ever get there.

EDIT: Also reading the rest of the thread I realize that the post that you linked out of context is slightly misleading (but I blame github's aggressive folding more than you here). Guido's answer comes after the following exchange:

stefantalpalaru: "Disregard Guido's objection. The "Python" trademark doesn't extend to "py2" or "py28". Read this for details: https://www.python.org/psf/trademarks/"

Guido: "Isn't the whole point that we're trying to solve this without lawyers?"

stefantalpalaru: "The whole point is that you've been sabotaging Python 2 for years and when someone does what needed to be done from the start, you come up with silly objections."

Guido: "OK, bring in the lawyers."

In that light, and given the other poster's ridiculously inflammatory take, Guido's answer seems rather level headed and appropriate IMO. He stands his ground, so to speak.

Re:Mercurial’s journey to and reflections on Python 3 by lizmat on Tuesday January 14, 2020 @06:56AM (#22043949)
Re: I understand your point of view but on the other hand we can make the parallel with Perl 5 and 6.

Please note that Perl 6 has been renamed to Raku (https://raku.org using the #rakulang tag on social media). So Perl and Raku are now considered to be different languages, albeit from the same inspiration.

Now, if Python 2 people would decide to rename Python 2 to something else, I guess it would be a mirrored parallel :-)

Re:Mercurial’s journey to and reflections on Python 3 by simias on Tuesday January 14, 2020 @09:06AM (#22044791)
That's precisely what's happening with the third-party Python2 forks. The renaming of Perl 6 occurred last October, specifically because of the problems caused by the confusion between the incompatible Perl 5 and 6 that caused a lot of trouble to the Perl people on either side for many years.

It's not a mirrored parallel, it's the Python folks learning from Perl's mistakes and making sure that this parallel won't come to be.

Re:Mercurial’s journey to and reflections on Python 3 by mjw1007 on Monday January 13, 2020 @02:12PM (#22037537)
I think it makes a great deal of sense for the Python core team to say "we're finished with Python 2 and want nothing more to do with it".

But I'm very disappointed that the Python Software Foundation isn't explictly supporting people who want to keep Python 2 compiling and running on modern systems. I think that would be well within their remit to "promote, protect, and advance the Python programming language".

This is particularly so because Python is widely used for scientific purposes, and being able to reproduce old results is valuable.

Even before Python 3.0 appeared, I came across scientists saying "I prefer to stick with Fortran because new Python versions break old code too frequently".

Re:Mercurial’s journey to and reflections on Python 3 by int_19h on Monday January 13, 2020 @02:37PM (#22037772)
PSF does not object to people who keep Python 2 compiling and running, such as ActiveState (https://www.activestate.com/company/press/press-releases/act...).

This case is different, because it's a project that uses the Python name, but actively adds features to the language. This is the classic example of brand confusion - someone might try to use it, find something to complain about, and PSF's reputation suffers as the result. They also get support overhead from the users of the fork (even if all they do is tell them to go away, that is still triage time that could be spent on other issues).

Re:Mercurial’s journey to and reflections on Python 3 by mjw1007 on Monday January 13, 2020 @03:10PM (#22038089)
"Does not object" is better than nothing, but I think it would be better if the PSF actively helped to coordinate this work (again, without bothering the core team). As far as I'm concerned, this is exactly the sort of thing that the PSF exists for.
Re:Mercurial’s journey to and reflections on Python 3 by forgotpwd16 on Tuesday January 14, 2020 @10:06AM (#22045330)
>This is particularly so because Python is widely used for scientific purposes, and being able to reproduce old results is valuable.

You can always download an old version and the respective libraries and use them to reproduce any results you want. That doesn't mean that old version should be supported anymore.

Re:Mercurial’s journey to and reflections on Python 3 by Klonoar on Monday January 13, 2020 @02:04PM (#22037465)
Longtime Python dev who was also annoyed by the 2 - 3 transition here.

I don't see Guido as in the wrong for that. It'd be a smack in the face when you spend years trying to finally push people to switch (for better or for worse) and then a project like this takes the SEO and gets to run freely with it.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Monday January 13, 2020 @02:09PM (#22037511)
Why should Guido or PSF get to tell people to stop using Python 2 even if they no longer want to work on it? It's ungraceful not to hand off maintainership on good terms to someone who wants to do the work.

Imagine if Stroustrup had done D and insisted that it be called C++ and wanted everyone to stop using the language everyone knew as C++ on Jan 1st 2020.

Re:Mercurial’s journey to and reflections on Python 3 by dragonwriter on Monday January 13, 2020 @02:22PM (#22037631)
> Why should Guido or PSF get to tell people to stop using Python 2 even if they no longer want to work on it?

They aren't stopping people from using Python 2, the language or Python 2, the software.

They are stopping people from using the name “Python” as the name of forked implementations of Python 2 not maintained by the PSF. No implementation not maintained by PSF is allowed to be called unqualified Python; the name is an important indicator of provenance. There are and have been plenty of third-party Python (2 and otherwise) implementations, the implementations just need their own names.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Monday January 13, 2020 @02:42PM (#22037822)
> They aren't stopping people from using Python 2, the language or Python 2, the software.

The effort to claim the binary name python for Python 3 is actively hostile to Python 3 and a thing that runs unmodified Python 2 unmodified on the same operating system installation. (It's unclear to me how much this is a PSF push, but at least the PEP isn't telling distros to refrain from this hostile-to-comaptibility action.)

> No implementation not maintained by PSF is allowed to be called unqualified Python

The best situation would be PSF hosting continued Python 2-compatible development by people who want to do the work.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Monday January 13, 2020 @03:10PM (#22038093)
> The best situation would be PSF hosting continued Python 2-compatible development by people who want to do the work.

For who? This costs the PSF manpower/overhead that they don't want to expend on a thing they don't want to maintain. It dilutes the language that the PSF are stewards of, and would further cause a schism in the python community. None of those things sounds good for python, its ecosystem, or the PSF. They sound good for, like, a few curmudgeonly companies and individuals that don't want to migrate.

I can't parse your first sentence, so I can't respond to it.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Monday January 13, 2020 @03:56PM (#22038475)
> For who?

For users of the Python 2 language who have a lot of Python 2 code and for whom migration doesn't make cost/benefit sense on technical merits of Python 3.

There's Tauthon. There's Active State's long-term support for Python 2. There's presumably Red Hat's long-term for Python 2. There are probably others. Also, there's the need to keep the server side of pip up and running for these to work.

It would be great if there was a common venue for collaboration for these by the parties who are interested in keeping Python 2 going. (I'm not suggesting that Python 3 core devs should do the work.) Like a foundation for Python software.

The first sentence meant that claiming the command-line executable name python for Python 3 is hostile to letting an execution environment for Python 3 and an execution environment for Python 2 co-exist going forward without having to modify existing programs that assume that python is for Python 2 and python3 is for Python 3.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Monday January 13, 2020 @04:33PM (#22038892)
> The first sentence meant that claiming the command-line executable name python for Python 3 is hostile to letting an execution environment for Python 3 and an execution environment for Python 2 co-exist going forward without having to modify existing programs that assume that python is for Python 2 and python3 is for Python 3.

Yes, but I don't believe I've seen any (real) suggestions to change PEP 394.

> There's Tauthon.

Which I claim is actively bad for python's ecosystem in the long term. It shouldn't be supported by any organization that wants what is best for Python.

> There are probably others. Also, there's the need to keep the server side of pip up and running for these to work.

That works just fine without any help. pypi continues to support python2 tags and wheels, and I doubt that'll change anytime soon.

> There's Active State's long-term support for Python 2. There's presumably Red Hat's long-term for Python 2.

So the entire reasonable bit here is that the PSF should provide something to help various enterprise companies manage backporting security patches. Which, like, I'm not sure what infrastructure is actually needed for that. They already make security patches public. Unless you're suggesting that LTS enterprise support offerings should co-ordinate additional feature work on python 2, which is both unusual and again I claim actively harmful to the ecosystem.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @12:43AM (#22042183)
> Unless you're suggesting that LTS enterprise support offerings should co-ordinate additional feature work on python 2, which is both unusual and again I claim actively harmful to the ecosystem.

If you have a large amount of Python 2 code that doesn't make sense to rewrite as Python 3 but does make sense to keep developing as opposed to just keep running as-is, it makes sense to want compatibility-preserving improvements to the language.

That such improvements are considered actively harmful comes from a point of view where there's a top-down imperative to shut down Python 2 in order to make Python 3 succeed. It's not harmful from the point of view of the code people have written in Python 2 being valuable.

The notion that there user community needs to work for Python (by porting to Python 3) and that Python 2 needs to be shut down as opposed to Python development valuing the existing code that had been developed is the core problem with Python 3.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Tuesday January 14, 2020 @01:45AM (#22042453)
> If you have a large amount of Python 2 code that doesn't make sense to rewrite as Python 3 but does make sense to keep developing as opposed to just keep running as-is, it makes sense to want compatibility-preserving improvements to the language.

But it really doesn't. If the new features are that valuable, you can convert your code. It's not actually that hard (I have a few 100kloc ported forward now, with millions of lines of dependencies that says so).

Re:Mercurial’s journey to and reflections on Python 3 by takeda on Monday January 13, 2020 @04:56PM (#22039153)
That project is not Python 2 though, it added features that made it incompatible with both Python 2 and Python 3. Just look at their effort to add wheel support.

Any project that forks changes name:

nagios -> icigna

mysql -> mariadb

NetBSD -> OpenBSD

FreeBSD -> DragonflyBSD

Python -> PyPy, Jython, IronPython

It would be crazy for them to keep the same name and not be compatible. It would cause confusion and also lead to increase of support tickets in wrong bug trackers.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @01:33AM (#22042397)
In those cases, the original project lived on. Here Python 3 is the incompatible fork, but because the technical fork is done by the folks who control the name and who want to shut the old thing down, so the compatible evolution of Python 2 had to change the name.
Re:Mercurial’s journey to and reflections on Python 3 by gjulianm on Monday January 13, 2020 @03:13PM (#22038122)
Your analogy is not appropriate. The actual situation with Tauthon is as if someone was not happy with C++17, so they forked C++14, added new features and changed syntax and then insisting to call it C++. It's just confusion for the users and it's in the best interest of the PSF to protect the Python name.
Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Monday January 13, 2020 @04:09PM (#22038611)
I'd agree if the Python core devs were still interested in evolving Python 2.x. But they aren't, so now no one else gets to do Python 2.8, either. It would be the best if the PSF provided a venue for Python 2.x development even if the folks who went on to do Python 3 weren't the people working on it.

Anyway, the core problem is a top-down effort to try to make a programming language of Python 2.x’s level of usage stop to the extent it’s stoppable under its license, because its creators wanted to do something else, as opposed to facilitating its user community to pool resources to continue its development. Does the PSF have a legal obligation to do such facilitation? No. Is the lack of such facilitation bad for parties who bought into Python when it was Python 2? Yes.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Monday January 13, 2020 @08:16PM (#22040789)
> core devs were still interested in evolving Python 2.x.

They absolutely are. In fact, python 3.9 is in the works right now, which has many new evolutions beyond 2.7.

You're arguing that the psf should treat python2 and 3 as different languages. In their (any my) opinion, this is harmful. It bifurcates python into two incompatible languages. That's bad long term (Perl).

In other words, what's best for python the language, and what's best for python2 the language are not the same. And for the psf, python is more important.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @01:50AM (#22042466)
> They absolutely are. In fact, python 3.9 is in the works right now, which has many new evolutions beyond 2.7.

I meant compatible (in the sense that old programs keep running and you can add new stuff to old programs using the new features) evolutions.

> You're arguing that the psf should treat python2 and 3 as different languages.

For practical purposes, they are different languages and the PSF has been treating them as distinct things.

> In their (any my) opinion, this is harmful. It bifurcates python into two incompatible languages. That's bad long term (Perl).

It indeed is bad. I hope that every other programming language community and designer takes a close look at what happened and makes sure never to do a Python 3 analog of their language.

> In other words, what's best for python the language, and what's best for python2 the language are not the same. And for the psf, python is more important.

That's the core problem from the perspective of Python 2 users. The organization that was the steward of the language that they invested in (in the form of writing code in the language) decided not only that a different programming language is more important for the org but that the old language needed to be shut down in order to benefit their new thing.

It's OK for people to get bored with a project and move onto something else, but with the level of usage that Python 2 had and has, it's very problematic for the language steward organization to turn around and seek to shut the language down instead of continuing to evolve it in a way that's respectful of the language users' investment in the language.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Tuesday January 14, 2020 @02:13AM (#22042556)
> a way that's respectful

You had like 10 years of warning and it's "disrespectful"? I don't think there's a chance of productivity if you're starting from that baseline level of entitlement. Sure, mandates are annoying. But I just can't fathom that.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @07:09AM (#22044029)
It's not about how many years of warning there was. It's about making users of the language to rewrite by mandate as opposed to the new features being incrementally adoptable into existing code bases. Sure, that means there are some language changes you never get to make.

Java, JavaScript, C, and C++, for example don't break investment in old code like Python 3 did. They form a reasonable baseline.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Tuesday January 14, 2020 @12:03PM (#22046755)
And we have kotlin, typescript, and rust due to those languages unwillingness to make breaking changes. The cpp committees unwillingness to remove old garbage from the language is iirc the most cited issue with the language by longtime users.

There are tradeoffs.

Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @04:37PM (#22049493)
You can add Kotlin to you app without rewriting all Java. You can add TypeScript to your app without rewriting all JavaScript. You can add Rust to your app with with rewriting all C++. Seems reasonable.

That Python 2 and 3 can't co-exist in an app is pretty bad in comparison.

Re:Mercurial’s journey to and reflections on Python 3 by joshuamorton on Tuesday January 14, 2020 @04:50PM (#22049600)
> That Python 2 and 3 can't co-exist in an app is pretty bad in comparison.

You're mistaken. I have python3 binaries and python2 binaries that share dependencies.

You're correct that fully automatic transpilation is impossible, but that doesn't mean that there can't be shared source. It does however mean that things like per-file flags or whatnot aren't possible. Python became a better language with text vs. bytes support, but that support couldn't be done in a backwards compatible way. Oh well.

> You can add Rust to your app with with rewriting all C++.

It's not as good as you seem to think. It's a nonstarter for a lot of people otherwise interested in adopting rust into existing codebases. Certainly not better than the py2/3 situation.

Kotlin interop also is troublesome, although granted better than rust/cpp or py2/3.

> That Python 2 and 3 can't co-exist in an app is pretty bad in comparison.

That python didn't get replaced by a different language is an incredible testament to the foresight of the python language stewards.

Re:Mercurial’s journey to and reflections on Python 3 by gjulianm on Monday January 13, 2020 @05:16PM (#22039331)
> as opposed to facilitating its user community to pool resources to continue its development

How does reusing the name facilitate the development? Every time there is a fork of an open-source project the name changes, precisely to avoid confusions. Reusing the Python name in a fork that is not just a redistribution, but a new version with new features and syntax, is just confusing, unusual and does not help anyone.

Re:Mercurial’s journey to and reflections on Python 3 by umanwizard on Monday January 13, 2020 @04:53PM (#22039126)
The difference is that the major C++14 implementations are still supported, and probably will be for as long as the concept of C++ exists.

There's no `--std=python2` flag you can pass to the interpreter, unfortunately.

Re:Mercurial’s journey to and reflections on Python 3 by gjulianm on Monday January 13, 2020 @05:12PM (#22039290)
There is no '--std=C++14-with-arbitrary-things-from-c++20' flag either, which is what this fork does. We can discuss whether breaking backwards compatibility was bad or necessary, but creating another fork of Python that backports some features of Python 3 is just adding confusion. If their primary purpose is supporting Python 2.7 applications, they can do just fine without calling it Python.
Re:Mercurial’s journey to and reflections on Python 3 by comex on Monday January 13, 2020 @06:30PM (#22039972)
There doesn’t need to be such a flag because C++17 was fully backwards compatible with 14, with some tiny exceptions nobody cares about.

Indeed, C++ has rarely made any breaking changes. A decade or so ago, GCC did cause some major ecosystem breakage, by cracking down on C++ constructs which had never been valid according to the spec but which GCC had previously allowed. When that happened, there was a flag to (at least partially) revert to the old behavior: -fpermissive.

Re:Mercurial’s journey to and reflections on Python 3 by Conan_Kudo on Monday January 13, 2020 @11:53PM (#22041963)
> There doesn’t need to be such a flag because C++17 was fully backwards compatible with 14, with some tiny exceptions nobody cares about.

This literally does not parse. How do you know "nobody" cares about those exceptions?

Re:Mercurial’s journey to and reflections on Python 3 by wott on Monday January 13, 2020 @08:08PM (#22040724)
'-std=C++14' already includes a few extensions, it is not pure C++14 but a superset. And then there is '-std=gnu++14' too.
Re:Mercurial’s journey to and reflections on Python 3 by CJefferson on Monday January 13, 2020 @07:19PM (#22040364)
No, this is like if someone was not happy with C++17 AND gcc removed its support for C++14. Instead, I can still happily compile C++14, C++11, C++03 and C++98 code with gcc.
Re:Mercurial’s journey to and reflections on Python 3 by hackbinary on Monday January 13, 2020 @02:09PM (#22037507)
I disagree. If someone else wants to continue to develop Python 2 outside the Python foundation and formalised development community, then that is their prerogative, but Python has the right to decide what is and is not Python.

Dilution of what is commonly accepted to be Python would not be a good thing, and would further add to confusion.

I know that platform upgrades are painful, but we need to move with the times or we'll all be mired in technical debt and old technology.

Re:Mercurial’s journey to and reflections on Python 3 by takeda on Monday January 13, 2020 @05:07PM (#22039249)
Yes, them keeping Python 2 alive for 10 years where Python 3 was developed caused a lot of issues, it would be extremely short sighted to allow third (incompatible) python into the mix.
Re:Mercurial’s journey to and reflections on Python 3 by hsivonen on Tuesday January 14, 2020 @01:52AM (#22042469)
> it would be extremely short sighted to allow third (incompatible) python into the mix

The whole point of Tauthon is that it is compatible with Python 2 (in the direction that old programs work).

Re:Mercurial’s journey to and reflections on Python 3 by camgunz on Monday January 13, 2020 @03:52PM (#22038426)
Letting "Python 2" zombie around is unacceptable. Python 3 is better in every way, and has been since 3.3 (which Armin deserves a lot of credit for).

Consider anyone who wants to build something with Python, whether it's a library, application, or service. What's better, having to build for Python 3 and 2, or just Python 3?

Thank God that Guido did this, despite knowing all the blowback he'd get. To me, that's super cool.

Re:Mercurial’s journey to and reflections on Python 3 by eesmith on Monday January 13, 2020 @04:08PM (#22038602)
"better in every way" ... except for 1) startup time (according to the linked-to article), 2) support for existing Python 2 code, and 3) support for Python 2 C extensions.

For example, https://blog.khinsen.net/posts/2017/11/16/a-plea-for-stabili... describes the "Molecular Modelling Toolkit (MMTK), which might well be the oldest domain-specific library of the SciPy ecosystem, will probably go away after 2020. Porting it to Python 3 is possible, of course, but an enormous effort (some details are in this Twitter thread[1]) for which resources (funding plus competent staff) are very difficult to find."

[1] The thread at https://twitter.com/khinsen/status/930749714567434240 includes "Lots of C modules written for Python 1.4 are waiting for enthusiastic code archeologists ;-)".

I don't think Hinsen is alone in that situation. I can well believe there are some people who, for example, plan to retire in about 5 years and would rather keep with with a Python 2 zombie than spend time to port working code to Python 3.

Re:Mercurial’s journey to and reflections on Python 3 by camgunz on Tuesday January 14, 2020 @02:00AM (#22042501)
Startup time is complex, but base startup only increased about 20ms, and that's being generous.

I'll admit Python 3 is still slower at a lot of things. But that feels like saying your new dog is even worse at math than your old one.

The C extension thing isn't Python's fault. It's the job of library and app authors to update. Do we complain that Vulkan has bad SunOS support? This is totally backwards.

Could Hinsen (and others) not just version their deps? It's not like people are erasing Python 2 off the internet. If his main worry is reproducibility, he should be doing that anyway.

---

I don't want to give the impression I like the whole Python 3 thing. I think it was a pretty big mistake and a huge missed opportunity. I'm very sympathetic to people who had to put in a lot of work for basically no good reason--Python 3 didn't really offer anything significantly better than 2 until... 3.5 (3.4 if you think the first pass at async was useful, I personally don't).

But I also find the ballyhooing about it really insufferable. Yeah it was a mistake; Armin Ronacher (as usual) was right. It was also over 11 years ago. Time to forget all about this and build cool stuff, please please please.

Re:Mercurial’s journey to and reflections on Python 3 by tick_tock_tick on Tuesday January 14, 2020 @05:17AM (#22043430)
It takes me 37ms to load hackernews a 20ms start time is embarrassing what is it even doing?
Re:Mercurial’s journey to and reflections on Python 3 by eesmith on Tuesday January 14, 2020 @05:40AM (#22043544)
It does a lot of module imports. Most of those probe the filesystem.

Try "python -vv -c 'pass'" - I'm only showing the first few dozen lines, and I've trimmed some of the paths for conciseness:

    % python -vv -c 'pass'
    import _frozen_importlib # frozen
    import _imp # builtin
    import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
    import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
    import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
    # installing zipimport hook
    import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
    # installed zipimport hook
    import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
    import '_io' # <class '_frozen_importlib.BuiltinImporter'>
    import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
    import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
    import _thread # previously loaded ('_thread')
    import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
    import _weakref # previously loaded ('_weakref')
    import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
    # miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/__init__.py
    # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc'
    # trying miniconda3/lib/python3.7/codecs.cpython-37m-darwin.so
    # trying miniconda3/lib/python3.7/codecs.abi3.so
    # trying miniconda3/lib/python3.7/codecs.so
    # trying miniconda3/lib/python3.7/codecs.py
    # miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc matches miniconda3/lib/python3.7/codecs.py
    # code object from 'miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc'
    import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
    import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44c90>
    # trying miniconda3/lib/python3.7/encodings/aliases.cpython-37m-darwin.so
    # trying miniconda3/lib/python3.7/encodings/aliases.abi3.so
    # trying miniconda3/lib/python3.7/encodings/aliases.so
    # trying miniconda3/lib/python3.7/encodings/aliases.py
    # miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/aliases.py
    # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc'
    import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd67d10>
    import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd440d0>
    # trying miniconda3/lib/python3.7/encodings/utf_8.cpython-37m-darwin.so
    # trying miniconda3/lib/python3.7/encodings/utf_8.abi3.so
    # trying miniconda3/lib/python3.7/encodings/utf_8.so
    # trying miniconda3/lib/python3.7/encodings/utf_8.py
    # miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/utf_8.py
    # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc'
    import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44bd0>
    import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
    # trying miniconda3/lib/python3.7/encodings/latin_1.cpython-37m-darwin.so
    # trying miniconda3/lib/python3.7/encodings/latin_1.abi3.so
    # trying miniconda3/lib/python3.7/encodings/latin_1.so
    # trying miniconda3/lib/python3.7/encodings/latin_1.py
    # miniconda3/lib/python3.7/encodings/__pycache__/latin_1.cpython-37.pyc
    matches miniconda3/lib/python3.7/encodings/latin_1
       ... many, many more lines omitted ...
This can be sped up a lot using a zipimport of the Python standard library, https://docs.python.org/3/library/zipimport.html?highlight=z... , where all of the standard library is put in a zipfile. Then there's a file access to get the zip metadata.

One of the things that bugged me in Python2 was that every startup imported UserDict:

    # trying python2.7/UserDict.so
    # trying python2.7/UserDictmodule.so
    # trying python2.7/UserDict.py
    # python2.7/UserDict.pyc matches python2.7/UserDict.py
    import UserDict # precompiled from python2.7/UserDict.pyc
This is because os.environ was an instance of UserDict:

    % python2.7 -c 'import os; print(os.environ.__class__.__bases__)'
    (<class UserDict.IterableUserDict at 0x1029b14c8>,)
Under Python3 this is spelled collections.abc.MutableMapping:

    % python3 -c 'import os; print(os.environ.__class__.__bases__)'
    (<class 'collections.abc.MutableMapping'>,)
which triggers its own set of imports:

    # trying python3.6/collections/abc.cpython-36m-darwin.so
    # trying python3.6/collections/abc.abi3.so
    # trying python3.6/collections/abc.so
    # trying python3.6/collections/abc.py
    # python3.6/collections/__pycache__/abc.cpython-36.pyc matches python3.6/collections/abc.py
    # code object from 'python3.6/collections/__pycache__/abc.cpython-36.pyc'
    import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x103c2ecf8>
There's better performance using SDD than a HDD, which is in turn better than using a networked filesystem.
Re:Mercurial’s journey to and reflections on Python 3 by tick_tock_tick on Tuesday January 14, 2020 @02:06PM (#22048132)
I'm guessing startup time just isn't an important goal for cython then? I've know they refused to implements some optimization that would significantly increase complexity but this seem like low hanging fruit?
Re:Mercurial’s journey to and reflections on Python 3 by eesmith on Tuesday January 14, 2020 @03:36PM (#22048993)
Oh, there's been plenty of work to reduce the Python startup cost.

It's just hard to fix.

I'm not sure the os.environ example I gave is low-hanging fruit now. The collections.abc module might be imported anyway.

This is neat! Python 3.7 added the `PYTHONPROFILEIMPORTTIME=1` environment variable to help track down these sorts of import overheads:

  % env PYTHONPROFILEIMPORTTIME=1 python -c pass
  import time: self [us] | cumulative | imported package
  import time:       523 |        523 | zipimport
  import time:       722 |        722 | _frozen_importlib_external
  import time:       156 |        156 |     _codecs
  import time:      2254 |       2409 |   codecs
  import time:      1293 |       1293 |   encodings.aliases
  import time:      7192 |      10893 | encodings
  import time:      1108 |       1108 | encodings.utf_8
  import time:       182 |        182 | _signal
  import time:      1069 |       1069 | encodings.latin_1
  import time:       395 |        395 |     _abc
  import time:      1486 |       1881 |   abc
  import time:      1540 |       3420 | io
  import time:       100 |        100 |       _stat
  import time:       975 |       1075 |     stat
  import time:      1481 |       1481 |       genericpath
  import time:      1734 |       3214 |     posixpath
  import time:      2558 |       2558 |     _collections_abc
  import time:      2234 |       9079 |   os
  import time:      1407 |       1407 |   _sitebuiltins
  import time:      3498 |       3498 |   sitecustomize
  import time:        85 |         85 |   usercustomize
  import time:      4129 |      18196 | site
Investigating further, the "import os" which triggered the UserDict/collections.abc is a consequence of "import site". If I use "python -S" then those aren't imported.
Re:Mercurial’s journey to and reflections on Python 3 by eesmith on Tuesday January 14, 2020 @05:21AM (#22043456)
I can't help but interpret your response as "Python 3 is better in every way ... for the ways I think are important."

For some of my programs, Python startup time is the main overhead. I avoid NumPy and SciPy if at all possible because they have a huge startup overhead.

Some of this is inherent in those packages. NumPy internally imports everything so someone can do "import numpy as np; np.package.subpackage.module.function()" without doing the intermediate imports.

This means NumPy is optimized for programmers (especially novice programmers) using NumPy in long-lived processes where startup cost is a negligible overhead.

Which isn't all use-cases for numeric computing.

15 years ago I supported a CGI-based web app. It was very important to pull out all the stops (delay imports until needed, use zip packages) because it was easier to do that than to re-write everything for another architecture.

The dog does count pretty well after all.

> It's the job of library and app authors to update.

Why? Linus Torvalds doesn't agree with you, for one.

As Hinsen points out,

] Unfortunately, the need for long-term stability is rather specific to scientific users, and not even all of them require it (see e.g. these two tweets by Titus Brown). So while Python 3 is probably a step forward for most Python users, it’s mostly a calamity for computational science.

Some scientific code has been able to run unchanged since the 1970s, through multiple new Fortran language releases.

Now, yes, I know the reasons for the changes to Python. I know the funding and organizational realities.

But why not recognize that for some situations Python 3 is not better?

Hinsen also comments on your proposal:

] The implication is that breaking changes in the infrastructure layers are OK and must be absorbed by the maintainers of layers 3 and 4. In view of what I just said about layer 4, it should be obvious that I don’t agree at all with this point of view. But even concerning layer 3, I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter.

> Could Hinsen (and others) not just version their deps?

He addresses that, I think. One of the other commenters gives a more complete reply at https://metarabbit.wordpress.com/2017/11/18/numpy-scipy-back... ending "Freezing the versions solves some problems, but does not solve the whole issue of backwards compatibility.".

> Time to forget all about this and build cool stuff, please please please.

I'll quote Hinsen again "I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter."

Your implicit statement is that mmtk (Hinsen's code base) isn't "cool stuff". Why? Simply because it's old, or because you don't know about it or need it? What other cool old stuff will die because it's part of a community without the resources to update?

Instead, accept that that loss is part of the trade-offs, be empathetic to those who suffer, and bear those lessons in mind for future work you do.

Re:Mercurial’s journey to and reflections on Python 3 by camgunz on Tuesday January 14, 2020 @03:48PM (#22049085)
Well, I'd like to start off by saying I think we agree overall. I do think Python 3's advantages didn't merit its disadvantages until many years after the initial release.

Second, I admit to engaging in hyperbole when I said "Python 3 is better in every way"; usually I'm on the other side of these, but I'm just so fed up with people complaining. But you're right, there are still ways Python 3 isn't "better". I'd love to have productive, technical discussions about them, but we can't seem to get beyond the "Python 3 was a super bad idea" stuff, and I'm totally uninterested in that.

But beyond that, you and I are mostly talking about different things. Python 3 isn't NumPy or SciPy. If you're building extensions on top of them, you need to look at their compatibility commitments. If you want them to make more commitments, you have to convince them. This isn't specific to software engineering; this is due diligence for anything you're gonna put years of work into.

Django's page [1] is a great example of this. Python has one too [2]. I don't have any idea about SciPy/NumPy; It looks like SciPy 1.2.0 was an LTS release supported until 1/1/2020, but what do I know.

But importantly, the end result of this "hey, do 100x the work otherwise our science won't be reproducible" stuff will be to force people out of producing free software for scientific computing. And the non-free stuff is expensive, good god. Surely this isn't what you want.

A better tactic here is to work with the developers in establishing more compatibility between releases. You probably aren't gonna get Fortran levels of compatibility--a language and platform that's seen very, very little change over the decades. But then again, the core selling point of scientific Python is that you get to use a modern platform with modern features. Asking for that along with a 50 year compatibility guarantee is a laughably tall order: you can't have it both ways without exponential amounts of work. So just like you're asking other engineers to be empathetic and respect your need for more compatibility with your extensions, you need to be more empathetic and respect their resources. And the best place to do that is probably their contact page [3], not Twitter, HN, or random blogs.

[1]: https://www.djangoproject.com/download/#supported-versions

[2]: https://devguide.python.org/#status-of-python-branches

[3]: https://www.scipy.org/scipylib/mailing-lists.html

Re:Mercurial’s journey to and reflections on Python 3 by eesmith on Wednesday January 15, 2020 @12:37AM (#22051991)
You write "we can't seem to get beyond the "Python 3 was a super bad idea" stuff, and I'm totally uninterested in that."

Perhaps your "fed up"-ness means you overlook conversations which do go beyond that? Or do you put me into that category as well?

> Python 3 isn't NumPy or SciPy. ... this is due diligence for anything you're gonna put years of work into.

Hinsen's essay discussed these issues related to "software layers and the lifecycle of digital scientific knowledge". He put Python in layer 1, and NumPy/Scipy in layer 2.

In his essay he also said "I would like to see the SciPy community define its point of view on these issues openly and clearly. ... It’s OK to say that the community’s priority is developing new features and that this leaves no resources for considering stability. But then please say openly and clearly that SciPy is a community for coding-intensive research and that people who don’t have the resources to adapt to breaking changes should look elsewhere. Say openly and clearly that reproducibility beyond a two-year timescale is not the SciPy community’s business, and that those who have such needs should look elsewhere."

So I'm not convinced that we are talking about different things as you are making points I already referred to, albeit indirectly.

I'm also not sure you understood all of Hinsen's points. I say this because you wrote ""hey, do 100x the work otherwise our science won't be reproducible" stuff"

But Hinsen said "Layer 4 code is the focus of the reproducible research movement" and "the best practices recommended for reproducible research can be summarized as “freeze and publish layer 4 code” -- a solution you mentioned earlier.

It's just that reproducibility isn't the only goal for stability.

Another is to be able to go back to a 15 year old project and keep working on it, without taking the hit of rewriting it to a new, albeit similar, language.

I also have a small amount of umbrage about your comment:

> So just like you're asking other engineers to be empathetic and respect your need for more compatibility with your extensions, you need to be more empathetic and respect their resources.

I earlier wrote "Now, yes, I know the reasons for the changes to Python. I know the funding and organizational realities."

Did you overlook that because of your '"fed up"-ness', or was that not enough for you?

Re:Mercurial’s journey to and reflections on Python 3 by camgunz on Wednesday January 15, 2020 @10:59AM (#22055325)
> Perhaps your "fed up"-ness means you overlook conversations which do go beyond that? Or do you put me into that category as well?

I do put you in that category, because you seem to be focused much more on the negative, rather than being constructive and trying to find solutions to problems.

> I'm also not sure you understood all of Hinsen's points. I say this because you wrote ""hey, do 100x the work otherwise our science won't be reproducible" stuff"

I've read and directly disagreed with his essay. His points are:

- Python 2 going away orphans a lot of software, because there's a lack of resources/willingness to port to Python 3.

- Python 3 didn't provide enough value to the scientific community to justify all the breakage (this is true for almost every community, btw).

- SciPy breaks compatibility roughly every 2-3 years, which is a bad fit for the pace of scientific computing.

- Beyond that, breaking compatibility threatens reproducibility.

- The SciPy community doesn't seem to know or care about compatibility concerns.

- Projects written on top of SciPy libraries ("Layer 3" code) have to keep updating, and they don't always have resources/willingness to do that.

- It would be cool if SciPy laid out a support schedule.

- It isn't cool that SciPy says, "hey use us", and then breaks compat all the time.

- There are some languages/platforms that haven't changed in decades, this isn't an excuse.

Here's what I've said:

- Agree Python 3 didn't provide enough value.

- If you want to build something on SciPy that you expect to last for decades, you should look for a compat guarantee. If you don't, that's on you.

- If you want new features plus decades of compat, that's a ludicrous amount of work.

- If you want to find a way forward, start a dialogue with SciPy devs.

Hinson's examples of Fortran and Java are illuminating. Fortran's a platform that's seen very minimal evolution over its history. That's exactly the reason people want to use SciPy instead of Fortran. Java's a platform with... billions of engineering hours? It's ironic that a guy who doesn't want to spend the resources to update his own software is asserting that someone else can continually deliver a modern scientific computing platform with new features while never breaking compat, they just don't feel like it ("It's all a matter of policy, not technology"). That's wrong, it's a question of resources.

---

My diagnosis here is communication breakdown. Everyone here wants the same thing: use a modern software stack for scientific computing. So again I'll say get on the mailing lists, get on IRC, go to the conferences, and talk to the engineers. Be constructive.

Re:Mercurial’s journey to and reflections on Python 3 by MiroF on Monday January 13, 2020 @06:25PM (#22039927)
Did you see the other replies on the thread?

Guido has absolutely every right here.

Re:Mercurial’s journey to and reflections on Python 3 by Shorel on Monday January 13, 2020 @04:24PM (#22038774)
I downgraded Deluge to the Python 2 version because the new one doesn't work in Windows and I use both operating systems.