This is incredibly insightful. I'm frankly a bit stunned. I would never have thought of this (and clearly the Python people also didn't) but it is absolutely true. It's also a very transferable strategy for any time compatibility is to be broken.
Is this an original thought of yours or do you have references to further reading?
OP wrote, note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.
You can - and people widely do - compile a CJS version of your library from ESM sources. You'll accept that there may be multiple copies of your library in the program but it does work. But in any case there's no reason to actually author in CJS, everything including the ESM shim can be generated from ESM.
A library author can provide cjs bindings to their library for as long as cjs remains relevant with the assumption that they won't use top level await.
Last time I checked they seemed to lean into using rollup to take care of the translation under the covers.
From what I have seen seems pretty typical to simply export the named properties using a fresh module.exports = {...} and then use default as the main export?
The (comparatively) few Python 2 projects that were alive by the release of Python 3 and still alive by the EOL of Python 2 could probably have been upgraded manually, rather than with a tool.
On the other hand, people not wanting to write new code in Python 3 because it would be incompatible with the Python 2 ecosystem – that was a real problem. That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.
If only people would have written new code in Python 3 right away, we would have been in a far better position by the time Python 2 would have been EOL'd, purely due to natural turnaround of code bases.
Porting Python 2 code to Python 3 was never the problem. Writing new code in Python 3 was.
You're, frankly, high as a kite. As the author of one such port on a large codebase, porting Python 2 code to Python 3 was absolutely a problem.
The issue with 2to3 is that it assumed you'd do the conversion as a one-shot and straight flip over with entirely separate codebases (or dropping Python 2 entirely) from there on, which is completely insane.
What the vast majority of transitioning projects needed was a way to migrate progressively through cross-version codebases:
* libraries weren't going to drop older Python versions, the very few which attempted that (I'm aware of dateutil at least) were extremely painful to users and IIRC ended up rolling back to a more progressive codebase
* programs being distributed were not going to drop Python 2 until EOF at least
* large codebases simply could not afford to perform a completely untested and uncheckable one-shot transition
That's why the community created cross-version support tools like six and friends, and lobbied hard for the reintroduction of cross-version features into Python 3 e.g. reintroduction of `callable` (3.2), reintroduction of the "u" prefix (3.3), reintroduction of % on bytes (3.5) and I'm sure others.
> Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so. […] Python 3 because it would be incompatible with the Python 2 ecosystem
What do you think "the python 2 ecosystem is" except for a ton of projects which needed to be ported from Python 2 to Python 3 exactly? "the vast majority of websites or data processing scripts" is not an ecosystem, tooling and libraries are.
Do you somehow think numpy and lxml and django just scrapped their entire Python 2 codebase and rewrote the entire thing from scratch? Of course not.
> That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.
3to2 would not have worked any better better than 2to3. The syntactic incompatibilities between the version were not the hard part, the semantic ones were, and 3to2 would not have succeeded any better than 2to3 there because it simply couldn't.
All 3to2 would have given you is a broken Python 2 codebase from a Python 3 codebase you could not even run.
https://news.ycombinator.com/newsguidelines.html
"Be kind. Don't be snarky. "
So for over a decade, you'd be writing Python 3 code but building a Python 2 project. After a decade, one would hopefully have managed to port everything over to Python 3, and can then drop the "compile to Python 2" step.
A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!
----
I don't see why 3to2 would not have worked. It's a compiler. We have written compilers before. We can write compilers. Compilers work. Both languages are Turing complete and about the same in expressiveness, so clearly compiling one to the other is something we can do.
What we cannot (reasonably) do is produce nice, human-looking target code from a compiler. This is what 2to3 attempted to do, and what a 3to2 would never need to do.
Who cares if the Python 2 code it produced looked a little wonky, as long as it was correct and roughly recogniseable as being built from the original Python 3 code? The Python 3 code is the one humans write and edit. The Python 2 code is just compiler output to make it compatible with all of the Python 2 ecosystem until one is ready to go fully Python 3.
----
This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.
Slowly but surely, more and more things are being written in the modern fashion, and eventually, when everything has caught up, the compatibility conversion to the legacy code can be dropped.
No. You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.
> A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!
It would half solve (at best) one of them.
> I don't see why 3to2 would not have worked. It's a compiler.
Unless your 3to2 would literally reimplement Python 3 in Python 2, it would be a translator / bridge. Which can't work due to the semantics difference between Python 2 and Python 3: not every change translates mechanically and reliably. If they did, 2to3 would have worked.
> Who cares if the Python 2 code it produced looked a little wonky,
The maintainer who has to debug, probably.
> as long as it was correct
Which it wouldn't be.
> This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.
The "modern way" of javascript is additive, it's about adding new features to the language, some of which can be implemented in terms of the old one (and which you can thus "backport" through a compiler or extending the existing APIs).
You don't run into pieces of code which are syntactically identical but semantically divergent. Which you absolutely do in P2 v P3.
Absolutely not. Well, you could, but the better approach would be to write individual modules in Python 3, and then compile those down to Python 2 at which point they can talk to the existing Python 2 code freely.
> Unless your 3to2 would literally reimplement Python 3 in Python 2
That's exactly what I'm suggesting. Take the existing Python 3 compiler and swap out the backend to generate Python 2 code instead of bytecode.
This is stuff people do every day for other languages. It is very far from rocket science.
It doesn't. Not that its job is easy, but its job is to take constructs which are brand new and translate them to an older version of the language.
Of the same language.
It doesn't have to deal with the language or the APIs literally working differently when used the same way.
The Python version of Babel would be 3to3 compiler.
The end result is something that can be Py3 native whenever I want. 2to3 is the right tool for the job. There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.
Oh but there is. If you can map them to bytecode, you can map them to Python 2, which is a fully capable language. It won't be a one-to-one map, but that's also not required.
Sure the generated code is a horrific mess to look at but even that's a solved problem with sourcemaps if you need to debug it.
the real issue is 2to3 didn't work so well, or exposed the exact issues why python 3's breaking changes were needed (80% sloppy bytes/str/unicode handling).
I’ll conclude with three guidelines for library authors to follow:
Provide a CJS version of your library
Provide a thin ESM wrapper for your CJS
Add an exports map to your package.json
</quote>
This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead. If ESM is the new way, library authors should instead have a CJS shim to prevent fossilization of cruft.