News for hackers. Stuff that matters.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by rjeli on Thursday August 06, 2020 @04:11AM (#24069032)
<quote>

I’ll conclude with three guidelines for library authors to follow:

Provide a CJS version of your library

Provide a thin ESM wrapper for your CJS

Add an exports map to your package.json

</quote>

This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead. If ESM is the new way, library authors should instead have a CJS shim to prevent fossilization of cruft.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kqr on Thursday August 06, 2020 @05:20AM (#24069354)
> This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead.

This is incredibly insightful. I'm frankly a bit stunned. I would never have thought of this (and clearly the Python people also didn't) but it is absolutely true. It's also a very transferable strategy for any time compatibility is to be broken.

Is this an original thought of yours or do you have references to further reading?

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by rjeli on Thursday August 06, 2020 @05:40AM (#24069461)
It's a common refrain when the py2/3 discussions get going. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by danmur on Thursday August 06, 2020 @06:16AM (#24069627)
This is something like what the future package and tools do, though I can say from experience it's not without its issues.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by desmap on Thursday August 06, 2020 @05:51AM (#24069517)
> should instead have a CJS shim

OP wrote, note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by Macha on Thursday August 06, 2020 @07:58AM (#24070155)
It's not possible to write a nice automated transparent one for arbitrary libraries because of top level await, but it's totally possible for the 95%.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by jkrems on Thursday August 06, 2020 @01:15PM (#24073075)
You cannot write a CJS shim for any library because the CJS shim cannot expose any ESM file with a synchronous API because importing ESM always returns a Promise.

You can - and people widely do - compile a CJS version of your library from ESM sources. You'll accept that there may be multiple copies of your library in the program but it does work. But in any case there's no reason to actually author in CJS, everything including the ESM shim can be generated from ESM.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by WorldMaker on Thursday August 06, 2020 @04:09PM (#24074922)
It is possible, but you have to hack the Node require() loader to do it, teach it a lot about ESM loading, and then hope the downstream CJS don't have timing bugs around "very long" synchronous require() loads.

https://github.com/standard-things/esm

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by jkrems on Thursday August 06, 2020 @08:31PM (#24076823)
I'm not sure that's really relevant here because esm is just one of the transpilers for module syntax, it's not running native modules at all. The same can be achieved with the babel CLI or TSC etc. and the right configuration. But it's not really "ESM loading". It's compiling ESM to CJS and then running CJS. It's true that transpiling to CJS will continue to work, no matter what.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by WorldMaker on Friday August 07, 2020 @12:06PM (#24083414)
To my understanding this ESM module uses native ESM loading on Node versions that support it, which is why others point out its behavior with top-level await is deficient in the same ways that Node's own behavior is, though it tries.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by desmap on Thursday August 06, 2020 @10:56AM (#24071459)
Re your point OP also commented/quoted someone with “I don’t think designing a system with the blanket assumption that some feature just won’t get used is a viable path.”
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by Macha on Thursday August 06, 2020 @12:25PM (#24072482)
Yes, someone cannot design a system to handle arbitrary libraries with the assumption.

A library author can provide cjs bindings to their library for as long as cjs remains relevant with the assumption that they won't use top level await.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by nightski on Thursday August 06, 2020 @11:43AM (#24071956)
Yea they are looking at the ecosystem as a whole. But if you are a library author you have the power to control these things.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by inquisitiveio on Thursday August 06, 2020 @10:59AM (#24071484)
I'm pretty sure services like pika.dev/skypack.dev/jspm.io are trying to solve the CJS to ESM problem so you can use it directly in the browser (or deno).

Last time I checked they seemed to lean into using rollup to take care of the translation under the covers.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by desmap on Thursday August 06, 2020 @11:02AM (#24071524)
That's not the point here. It's about that a thin CJS wrapper to an ESM module isn't possible but the way around is.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by inquisitiveio on Thursday August 06, 2020 @12:06PM (#24072256)
Sorry I misread your comment. But is it not easier to write an CJS wrapper around ESM given that its format is a lot looser?

From what I have seen seems pretty typical to simply export the named properties using a fresh module.exports = {...} and then use default as the main export?

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by 1337shadow on Thursday August 06, 2020 @04:15AM (#24069050)
2to3 was still useful because all code was in python 2 when python 3 first came out.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kqr on Thursday August 06, 2020 @05:19AM (#24069348)
Sure, but maybe upgrading the existing Python 2 code wasn't the problem: the vast majority of the Python 2 code alive when Python 3 was released was probably scrapped by the time Python 2 was EOL'd. Over a decade is a very long time for a codebase to live. (Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so.)

The (comparatively) few Python 2 projects that were alive by the release of Python 3 and still alive by the EOL of Python 2 could probably have been upgraded manually, rather than with a tool.

On the other hand, people not wanting to write new code in Python 3 because it would be incompatible with the Python 2 ecosystem – that was a real problem. That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.

If only people would have written new code in Python 3 right away, we would have been in a far better position by the time Python 2 would have been EOL'd, purely due to natural turnaround of code bases.

Porting Python 2 code to Python 3 was never the problem. Writing new code in Python 3 was.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by masklinn on Thursday August 06, 2020 @06:10AM (#24069601)
> Porting Python 2 code to Python 3 was never the problem.

You're, frankly, high as a kite. As the author of one such port on a large codebase, porting Python 2 code to Python 3 was absolutely a problem.

The issue with 2to3 is that it assumed you'd do the conversion as a one-shot and straight flip over with entirely separate codebases (or dropping Python 2 entirely) from there on, which is completely insane.

What the vast majority of transitioning projects needed was a way to migrate progressively through cross-version codebases:

* libraries weren't going to drop older Python versions, the very few which attempted that (I'm aware of dateutil at least) were extremely painful to users and IIRC ended up rolling back to a more progressive codebase

* programs being distributed were not going to drop Python 2 until EOF at least

* large codebases simply could not afford to perform a completely untested and uncheckable one-shot transition

That's why the community created cross-version support tools like six and friends, and lobbied hard for the reintroduction of cross-version features into Python 3 e.g. reintroduction of `callable` (3.2), reintroduction of the "u" prefix (3.3), reintroduction of % on bytes (3.5) and I'm sure others.

> Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so. […] Python 3 because it would be incompatible with the Python 2 ecosystem

What do you think "the python 2 ecosystem is" except for a ton of projects which needed to be ported from Python 2 to Python 3 exactly? "the vast majority of websites or data processing scripts" is not an ecosystem, tooling and libraries are.

Do you somehow think numpy and lxml and django just scrapped their entire Python 2 codebase and rewrote the entire thing from scratch? Of course not.

> That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.

3to2 would not have worked any better better than 2to3. The syntactic incompatibilities between the version were not the hard part, the semantic ones were, and 3to2 would not have succeeded any better than 2to3 there because it simply couldn't.

All 3to2 would have given you is a broken Python 2 codebase from a Python 3 codebase you could not even run.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by nailer on Thursday August 06, 2020 @07:57AM (#24070140)
> You're, frankly, high as a kite.

https://news.ycombinator.com/newsguidelines.html

"Be kind. Don't be snarky. "

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kqr on Thursday August 06, 2020 @08:17AM (#24070245)
Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do? You'd write new code (and incrementally port old code) to Python 3, and then when you build the project you'd compile the Python 3 code down to Python 2 before building the entire project as Python 2, including all dependencies.

So for over a decade, you'd be writing Python 3 code but building a Python 2 project. After a decade, one would hopefully have managed to port everything over to Python 3, and can then drop the "compile to Python 2" step.

A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!

----

I don't see why 3to2 would not have worked. It's a compiler. We have written compilers before. We can write compilers. Compilers work. Both languages are Turing complete and about the same in expressiveness, so clearly compiling one to the other is something we can do.

What we cannot (reasonably) do is produce nice, human-looking target code from a compiler. This is what 2to3 attempted to do, and what a 3to2 would never need to do.

Who cares if the Python 2 code it produced looked a little wonky, as long as it was correct and roughly recogniseable as being built from the original Python 3 code? The Python 3 code is the one humans write and edit. The Python 2 code is just compiler output to make it compatible with all of the Python 2 ecosystem until one is ready to go fully Python 3.

----

This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.

Slowly but surely, more and more things are being written in the modern fashion, and eventually, when everything has caught up, the compatibility conversion to the legacy code can be dropped.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by masklinn on Thursday August 06, 2020 @11:40AM (#24071924)
> Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do?

No. You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.

> A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!

It would half solve (at best) one of them.

> I don't see why 3to2 would not have worked. It's a compiler.

Unless your 3to2 would literally reimplement Python 3 in Python 2, it would be a translator / bridge. Which can't work due to the semantics difference between Python 2 and Python 3: not every change translates mechanically and reliably. If they did, 2to3 would have worked.

> Who cares if the Python 2 code it produced looked a little wonky,

The maintainer who has to debug, probably.

> as long as it was correct

Which it wouldn't be.

> This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.

The "modern way" of javascript is additive, it's about adding new features to the language, some of which can be implemented in terms of the old one (and which you can thus "backport" through a compiler or extending the existing APIs).

You don't run into pieces of code which are syntactically identical but semantically divergent. Which you absolutely do in P2 v P3.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kqr on Thursday August 06, 2020 @12:40PM (#24072649)
> You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.

Absolutely not. Well, you could, but the better approach would be to write individual modules in Python 3, and then compile those down to Python 2 at which point they can talk to the existing Python 2 code freely.

> Unless your 3to2 would literally reimplement Python 3 in Python 2

That's exactly what I'm suggesting. Take the existing Python 3 compiler and swap out the backend to generate Python 2 code instead of bytecode.

This is stuff people do every day for other languages. It is very far from rocket science.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by chrisweekly on Thursday August 06, 2020 @11:22AM (#24071733)
Amen. Babel gets a lot of things right (and arguably has a tougher job than a 3to2 compiler would, given the speed of change and the target breadth / problem surface area).
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by masklinn on Thursday August 06, 2020 @11:43AM (#24071960)
> arguably has a tougher job than a 3to2 compiler would

It doesn't. Not that its job is easy, but its job is to take constructs which are brand new and translate them to an older version of the language.

Of the same language.

It doesn't have to deal with the language or the APIs literally working differently when used the same way.

The Python version of Babel would be 3to3 compiler.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kqr on Thursday August 06, 2020 @12:43PM (#24072678)
Babel is not limited to JavaScript frontends. You can take any language and write a Babel front-end for it. This is, for example, how Fable compiles F# code to JavaScript. Those two languages are very different from each other -- far more differences than Python 2 and Python 3 -- and it works brilliantly.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kevin_thibedeau on Thursday August 06, 2020 @11:53AM (#24072095)
I made all my Py2 code forward compatible with 2to3. It was mostly a smooth process when you're using the future imports and know what landmines to avoid. It still required manual intervention at times with Unicode support and libraries that have breaking API changes between versions.

The end result is something that can be Py3 native whenever I want. 2to3 is the right tool for the job. There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kqr on Thursday August 06, 2020 @12:55PM (#24072835)
> There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.

Oh but there is. If you can map them to bytecode, you can map them to Python 2, which is a fully capable language. It won't be a one-to-one map, but that's also not required.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by kevin_thibedeau on Thursday August 06, 2020 @01:26PM (#24073191)
Not going to work for async.
Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by Macha on Thursday August 06, 2020 @01:32PM (#24073275)
Why not? Provide your own future implementation, transform the async code into a state machine. Babel does exactly this for JavaScript async.

Sure the generated code is a horrific mess to look at but even that's a solved problem with sourcemaps if you need to debug it.

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by guitarbill on Thursday August 06, 2020 @06:16AM (#24069628)
good theory, doesn't align with my (limited) experience. writing new code to be both 2 and 3 compatible was quite possible and not too tricky. six helped, `from __future__` imports, etc. however, everywhere i've worked still had some python 2 only code running - even to this day unfortunately.

the real issue is 2to3 didn't work so well, or exposed the exact issues why python 3's breaking changes were needed (80% sloppy bytes/str/unicode handling).

Re:Node Modules at War: Why CommonJS and ES Modules Can’t Ge... by nurettin on Thursday August 06, 2020 @08:35AM (#24070341)
PSF wanted people to migrate to python3 for reasons including not wanting to support multiple languages. Why would they write some tool to convert new python3 code back to python2 that would ultimately work against their agenda? It doesn't make sense to me.