> just zipped the whole folder node_modules and all and shipped that

I mean, in a sense, something similar is what you should do if you're aiming for maximum reproducibility, of course, excluding the things that you don't need.

For a comparison, if you wanted to build a Docker container, it's going to include the node_modules folder in it, instead of installing dependencies on startup. So, in a word, somewhat similar to a fat .jar file (e.g. that has all dependencies, vs assuming that the app server will provide some).

My problem is that sometimes it's not awfully clear how to exclude all of the stuff that you don't need.

For example, my homepage is made with Ruby on Rails, for which I have a basic container image. But to actually bundle all of the resources, I do need Python, Node (with Yarn in my case) and also a bunch of gems, so I also make a separate dev container image.

I can install all of the dependencies I need for packaging and run them in a multi-stage image, carrying over the packaged files to something based on the basic image for just running it (e.g. without Python and Node), but I still need the gems to be present and there's no obvious way to carry over only the ones I need for running a Rails app.

Even without containers, this very same pain is present in many tech stacks, even if you wanted to run software directly on the servers to which you deploy it - many stacks out there aren't really big on fully self-contained executable packages (or even ones that depend on a particular version of a runtime, like JVM or .NET, but have everything else included).

Ergo, working with Ruby, Python, Node and many others can be a bit more troublesome than it should be. Personally, containers alleviate some of those problems to a degree, at the expense of exposing that complexity to you during build time.

Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step. I'm doing it, works like a charm. It'd be really weird and wasteful to ship my TypeScript and Eslint and whatever to cloud/Lambda/...

> Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step.

This is pretty close to the solution and would work if I ran just Node.js server side. Then I could easily have separate sets of dependencies for development and deployment/running, which is what most folks should do. Yet in certain tech stacks, things aren't as easy and there is reliance on system folders instead of local ones.

For example, my application that's running on the server should have absolutely nothing to do with Node, since it should run Ruby alone and use the bundled files, however it should have all of the Ruby Gems (basically packages) available. Of course, I also need Node and possibly something else during build time, for generating my assets.

For example, my intermediate container images might look like:

  FROM ruby_with_node_and_python AS builder

  # Copy only what's needed for installing dependencies and other garbage (e.g. changed code files won't invalidate Docker cache)
  COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app

  # install front end dependencies
  RUN yarn install
  # add code and run asset creation
  COPY ./src /app
  RUN bundle exec rake assets:clobber RAILS_ENV=production
  RUN bundle exec rake assets:precompile RAILS_ENV=production

  # install back end dependencies
  RUN bundle install

  # run tests (local SQLite, for example)
  RUN rails db:migrate RAILS_ENV=test
  RUN rails test RAILS_ENV=test

  # clean up files after build (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too)
  RUN rm -rf src/node_modules
  # clean up files after test (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too)
  RUN rm -rf src/log/*.log src/tmp/*

And then my container images for deployment would look like this:

  FROM ruby_with_nothing_else AS runner

  # copy over application files
  COPY --from=builder /app /app

Except that this doesn't work! Why? Because Ruby Gems aren't installed in the application directory, but instead sit in a variety of directories on the actual file system. So instead I'd need something like:

  FROM ruby_with_package_manager AS runner

  # we actually could not carry over the precompiled gems etc. because I'm stupid, so instead we now run the install again :(
  # well it's not like you need front end files here, but this doesn't really mean much
  COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app

  # install back end dependencies
  RUN bundle install

  # copy over application files
  COPY --from=builder /app /app

So while npm/yarn allows you to avoid some of the issues with `npm install --production` to exclude the devDependencies, other tech stacks aren't quite there yet. In addition, it feels kind of crazy to ship package managers like npm/yarn/pip/composer/bundle/maven or anything else in containers that should be immutable, so in practice you'd end up with more stages, even if you knew how to carry over the packages from one image to another in the technology stacks where it's not entirely clear how to do that.

I actually tried something similar to this, but it simply didn't work, especially when native extensions were needed for some of the Gems: https://stackoverflow.com/a/33778136

yeah that's the biggest issue with node/npm. 90% of your code lives on the registry, and you probably not even sure what 99% of it actually does.

would be nice if you could somehow reverse the compilation of your build back into a source folder, instead of having node_modules - but i think that's probably not possible.

Isn't that exactly what bundling does?

i think bundling generates your build from the source, but i mean generating a source from a build. so what i mean is more like reverse bundling.

so that you don't have to pull node_modules every time and install. i;m not totally sure but i dont think you can turn a minified bundled js files back into their original node_module source files...

That very much depends on the build -- if you've a source map, then it may very well be possible. And while I know not everyone publishes them, I strongly recommend doing so.

But honestly, ship your sources with your generated files. Please.

You can use esbuild to produce a single file for the Lambda handler. That’s the approach taken by AWS CDK via NodejsFunction construct.