News for hackers. Stuff that matters.
Re:Warp – self-contained, single binary applications by w8rbt on Friday October 12, 2018 @04:34PM (#18204510)
Very neat. Would like to see more languages targeting single binaries. This is one reason (I believe) that has made Go so popular.
Re:Warp – self-contained, single binary applications by rkeene2 on Friday October 12, 2018 @05:28PM (#18204844)
Tcl is still the king of this kind of distribution with Starkits/Starpacks. Since you can virtualize the filesystem within Tcl, you can pack in all your resources as regular files and generally scripts will work without knowing that this has happened (and without hacky extracting of the archive to disk before running, or while running).
Re:Warp – self-contained, single binary applications by paavoova on Friday October 12, 2018 @10:38PM (#18206082)
Python can do something similar albeit far simpler via packaging as an executable zipfile. Just structure everything you need in a directory and zip up with __main__.py in the root. All resources are available as an in-memory binary stream, including individual source .py files. Of course if you want to package something like a C library, you need to unpack that to disk to be able to access it with ctypes.
Re:Warp – self-contained, single binary applications by stdclass on Saturday October 13, 2018 @02:13AM (#18206691)
PHP has something similar, called .phar https://en.wikipedia.org/wiki/PHAR_(file_format)
Re:Warp – self-contained, single binary applications by creatornator on Saturday October 13, 2018 @01:12AM (#18206545)
Are you referring to the zipapp library? I had no idea this even existed until now but that seems pretty clever. Huh. Why can't it also dynamically unzip a shared object library?
Re:Warp – self-contained, single binary applications by paavoova on Saturday October 13, 2018 @02:08AM (#18206678)
You don't need to use any libraries, you can do something as simple as:

  $ zip -r mymodule.zip mymodule/
  $ echo '#!/usr/bin/env python3' > myapp
  $ cat mymodule.zip >> myapp
  $ chmod 755 myapp
CPython will automatically detect it's a zipfile, unzip it in-memory, and import the __main__.py in the root of the module and any nested modules.

The reason why you have to handle shared libraries and any other non-.py data yourself is because Python doesn't know what to do with it. You can access it as binary data via pkgutil.get_data(), but linking shared libraries is system-dependent and Python doesn't load them itself. As the dynamic linker can't find a shared library in a zip file, the only thing you can do is extract it separately. This is documented in https://docs.python.org/3/library/zipimport.html

Any files may be present in the ZIP archive, but only files .py and .pyc are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed.

Re:Warp – self-contained, single binary applications by therein on Saturday October 13, 2018 @01:40AM (#18206619)
He's talking about pex. [0]

[0] https://github.com/pantsbuild/pex

Re:Warp – self-contained, single binary applications by pmorici on Saturday October 13, 2018 @02:21AM (#18206720)
You can simulate a single file exe using py2exe and NSIS. That doesn't have the no library limitation.
Re:Warp – self-contained, single binary applications by dev_dull on Friday October 12, 2018 @06:33PM (#18205198)
This is an awesome design concept and I'm surprised other languages aren't copying it.
Re:Warp – self-contained, single binary applications by rkeene2 on Saturday October 13, 2018 @04:38PM (#18209891)
It generally takes other languages a long time to become more like Tcl ;-)
Re:Warp – self-contained, single binary applications by ASalazarMX on Friday October 12, 2018 @06:32PM (#18205192)
I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here).

It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed all applications.

I'd rather be in the latter than the former situation when the next vulnerability happens.

Re:Warp – self-contained, single binary applications by nur0n on Friday October 12, 2018 @07:08PM (#18205354)
I think the inverse argument can be made against shared libraries: if an update introduces a vulnerability, now all programs which depend on that library become vulnerable.
Re:Warp – self-contained, single binary applications by flingo on Saturday October 13, 2018 @07:46AM (#18207533)
Security problems often exist for years at a time.

Rather than making all of those applications vulnerable at the same time, they slowly become vulnerable as the release binaries are linked against bugged code. If it's not linked at runtime, or recompiled, it'll be vulnerable forever.

Re:Warp – self-contained, single binary applications by therein on Saturday October 13, 2018 @01:42AM (#18206624)
I would like to think that projects overall tend to fix more bugs than introduce them. It's not like projects go from orderly to disorderly over time.
Re:Warp – self-contained, single binary applications by quickben on Saturday October 13, 2018 @02:02AM (#18206667)
But some are high value targets (crypto ones, etc).
Re:Warp – self-contained, single binary applications by antaviana on Saturday October 13, 2018 @03:20AM (#18206848)
If I write malware, I guess it’s possibly easier to infect one library than thousands of executables.
Re:Warp – self-contained, single binary applications by Izkata on Monday October 15, 2018 @12:09PM (#18220915)
> but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here).

I believe you're referring to an OpenSSH vulnerability?

...though, I did a search just now to find when it was from (I remember there being a major one around the time you're thinking of), and instead found posts from only two months ago about another one that has existed for at least 18 years.

Re:Warp – self-contained, single binary applications by TimTheTinker on Friday October 12, 2018 @07:01PM (#18205326)
I think this represents a fundamental (and rather annoying) difference between Linux and Windows.

On Windows, the only platform-supported global library format is COM, and that requires a very specific programming style--HRESULTs and all the rest. True, registered COM DLLs can be linked to without separate header/symbol files, and installing new programs/libraries never requires local compilation, but the extra overhead of COM itself, and the fact that it's a walled garden code-wise, makes it rather unpopular. It's also nigh on impossible to port COM code to other apps without resorting to something like WINE (which can't compete performance-wise when running highly parallel COM apps/services). Not to mention how ugly C++ COM source files are.

It would be so nice for us all if Windows shipped with compilers, straight-up C libraries with headers, a package manager, a better shell, ...

Re:Warp – self-contained, single binary applications by TimJYoung on Friday October 12, 2018 @08:24PM (#18205644)
Windows has C-style, shared libraries just like Linux - DLLs. The entire Win32 API that is the backbone of Windows is implemented in system DLLs.

Every compiler that produces native binaries for Windows links with these DLLs (C, C++, Go, FreePascal/Delphi, etc.), either statically or dynamically, using C calling conventions.

COM is the Component Object Model, which is implemented in DLLs, but is something different:

https://en.wikipedia.org/wiki/Component_Object_Model

COM, or some form of it, is used heavily with .NET and the new UWP runtime, but those are newer runtimes that sort of sit on top of the Win32 API (to some extent, UWP is actually integrated into the OS itself, but, AFAIK, still uses the underlying Win32 APIs).

Re:Warp – self-contained, single binary applications by TimTheTinker on Monday October 15, 2018 @07:54PM (#18225193)
Thanks for the corrections.
Re:Warp – self-contained, single binary applications by ComputerGuru on Saturday October 13, 2018 @01:51AM (#18206649)
You sound extremely confused. COM has not been the only global dependency format on Windows since.. longer than I can remember. Heck, even when .NET first shipped in 2000 or 2001 you could register .NET binaries in the GAC. And any DLL in system32 is automatically available for global consumption.
Re:Warp – self-contained, single binary applications by finchisko on Saturday October 13, 2018 @01:27AM (#18206584)
I think. You can have global DLLs in Windows. They just need to be in PATH.
Re:Warp – self-contained, single binary applications by arendtio on Friday October 12, 2018 @05:30PM (#18204851)
By the way, if you are looking for a tool to embed resource files in Go binaries you might find these interesting:

- https://github.com/benbjohnson/genesis

- https://github.com/go-bindata/go-bindata

Re:Warp – self-contained, single binary applications by abright on Saturday October 13, 2018 @10:02AM (#18207974)
I've not used those two but have always had success with

- https://github.com/gobuffalo/packr

I like that it allows you to `go run` without changing any code.

Re:Warp – self-contained, single binary applications by artursapek on Friday October 12, 2018 @05:47PM (#18204954)
We use go-bindata a lot - love that package
Re:Warp – self-contained, single binary applications by munk-a on Friday October 12, 2018 @05:08PM (#18204737)
It is nice to have the option to go with either path. I think it's a bit less relevant in the modern development world but it can also be nice to leverage dynamic linking to present several executable entry points that share a large pool of common logic between them.
Re:Warp – self-contained, single binary applications by dinglejungle on Friday October 12, 2018 @09:45PM (#18205915)
Re:Warp – self-contained, single binary applications by jpic on Friday October 12, 2018 @04:52PM (#18204634)
I wish <my favorite language> also had this. However, containers work. Not sure why they do java though, i thought java has been happy with jars so far.
Re:Warp – self-contained, single binary applications by regecks on Friday October 12, 2018 @05:06PM (#18204727)
If anybody is thinking about Java, check Graal out. I was able to successfully create fast, completely static ELF binaries of real JVM applications from fat JARs (not in the fake self-executable archive way).

It's still early days in terms of what's available in the runtime (e.g. the AWT subsystem is missing), but it's pretty damn impressive anyway.

Re:Warp – self-contained, single binary applications by meddlepal on Friday October 12, 2018 @05:15PM (#18204776)
JARs are fine until you need to get someone to install a JRE. Not a problem for server apps, but definitely not painless for client distributed apps.
Re:Warp – self-contained, single binary applications by pjmlp on Friday October 12, 2018 @05:39PM (#18204906)
You can bundle everything together, or use a commercial JDK with AOT compilation support.
Re:Warp – self-contained, single binary applications by meddlepal on Friday October 12, 2018 @07:43PM (#18205474)
Neither of those options are "painless".
Re:Warp – self-contained, single binary applications by sebazzz on Sunday October 14, 2018 @07:33AM (#18212426)
Is it? Often you would distribute an installer anyway, right?
Re:Warp – self-contained, single binary applications by weberc2 on Friday October 12, 2018 @05:13PM (#18204762)
We're struggling using containers for our Python and Javascript applications. I think we're using venvs and node_modules and we oughtn't (or we shouldn't keep them in our project directory where they sometimes--but not always--get overwritten by source code volume mounts).
Re:Warp – self-contained, single binary applications by TheDong on Friday October 12, 2018 @06:24PM (#18205149)
It can be a little tricky to do things right here.

There are a few things of note:

1. The .dockerignore file can be used to prevent use of your `node_modules` folder during `docker build`, even if you have something like `COPY . .` in your Dockerfile.. This can let you create a new `node_modules` folder from your lockfile as part of the docker build process to create an image for testing/deployment.

2. You can maintain separate Dockerfiles and such for development and for release, so e.g. for development you might use volumes and not copy things in, but for release you wouldn't use volumes for source code.

I can't tell from what you said, but it seems like one of those two tips might be relevant.

It's okay to have a development setup which doesn't use containers and then use containers for deployment (so long as you have an integ or staging environment that uses containers) as well. It's quite reasonable to have a venv during development, but inside the docker image to not use a venv at all since things will already be reasonably isolated inside the container's fs.

Re:Warp – self-contained, single binary applications by weberc2 on Friday October 12, 2018 @07:42PM (#18205468)
Thanks for the suggestions! I had to check, but we do both of those things. To make sure that node_modules aren't overwritten, we mount an empty named volume to the node_modules directory. I guess mounting the named volume directory somehow causes the node_modules directory from the base image to appear inside of the bind-mounted directory. We do the same thing for venvs as well.

We use venvs inside of the Docker container because we use pipenv and apparently pipenv's support for installing to the system is buggy and/or idiosyncratic.

Re:Warp – self-contained, single binary applications by TheDong on Friday October 12, 2018 @09:51PM (#18205935)
Named volumes are hacky to no end and you shouldn't be using them to make sure it's empty; you should be using `.dockerignore` or being careful about what you copy in.

The pipenv bit is probably more reasonable, and I'm afraid I haven't used it enough to be sure what rough edges are likely there.

Re:Warp – self-contained, single binary applications by devxpy on Saturday October 13, 2018 @03:22AM (#18206852)
I've been using shiv recently for my django deploys.

You might wanna check it out.

Here is a small readme I put together...

https://github.com/devxpy/shiv/blob/8d8298d21380dcf0b1970856...

Re:Warp – self-contained, single binary applications by dharmab on Friday October 12, 2018 @05:38PM (#18204898)
Re:Warp – self-contained, single binary applications by weberc2 on Friday October 12, 2018 @05:49PM (#18204969)
Sorry if I'm a little dense; that doesn't seem like it solves any problems. It just says "use virtualenv and/or docker". I guess I was hoping for "how to manage dependencies in a localdev-friendly way for a Python/Docker app" or something.
Re:Warp – self-contained, single binary applications by VonGallifrey on Friday October 12, 2018 @06:44PM (#18205250)
Not sure if that helps, but here is what I do:

We just use a requirements.txt [1] for each service and run pip with the -r flag in the dockerfile. No Virtualenv in the container.

Most of the time we run these containers with docker-compose locally, but sometimes we want to run the service outside the container. For that we create a virtualenv outside the folder where we keep the source for the service. The reason for that is that we don't want to accidental copy the virtualenv in to the container. It wouldn't do much there but it would increase the container size.

You can do this easily with just "python3 -m venv", but there are some tools that help with that as well. I personally use pyenv-virtualenv [2] which just keeps all virtualenvs in ~/.pyenv/versions/<env>, but there is also conda [3] and virtualenvwrapper [4] which also store the virtualenvs in a central directory.

I am not sure if there really is any more to it.

[1]: https://pip.pypa.io/en/stable/user_guide/#requirements-files

[2]: https://github.com/pyenv/pyenv-virtualenv

[3]: https://conda.io/docs/user-guide/getting-started.html#managi...

[4]: https://virtualenvwrapper.readthedocs.io/en/latest/

Re:Warp – self-contained, single binary applications by dsnuh on Friday October 12, 2018 @05:59PM (#18205039)
I'm not sure if I understand correctly. Are you saying that you run venv commands as part of the build step for your containers (i.e. a RUN line in you Dockerfile) and then pip installing modules in another build step?
Re:Warp – self-contained, single binary applications by weberc2 on Friday October 12, 2018 @07:36PM (#18205449)
No, we use pipenv instead of pip, and pipenv manages a venv. There are `--system` flags, but I guess support for installing things to the system is a little buggy or idiosyncratic or something so we use the venv behavior. I'm not sure why we tell pipenv to install the venv in the project directory though. We do the pipenv install as a RUN line in our Dockerfile. Does this answer your question?
Re:Warp – self-contained, single binary applications by dsnuh on Friday October 12, 2018 @07:45PM (#18205482)
Yeah, I think I have an understanding now. We use tagged Docker base images with the app dependencies baked in so you can just use a FROM your Dockerfile and know you have the dependencies in the image and you just add and run your app code.
Re:Warp – self-contained, single binary applications by weberc2 on Friday October 12, 2018 @07:49PM (#18205500)
Yeah, we do the same thing. This seems to work pretty well, except right now we build these images out of band of our normal build process (if you change these base images, you're expected to kick off another CI job that builds/pushes new versions and updates the versions in the FROM statements in our production Dockerfiles (and our docker-compose.ymls). I'm not sure why we're doing this, since the build cache is actually pretty good at avoiding unnecessary base image builds.
Re:Warp – self-contained, single binary applications by dsnuh on Friday October 12, 2018 @09:02PM (#18205779)
That's how we do it also. You need to integrate a new process that is triggered on base image rebuild that redeploys your container with the new base image (which has security fixes, etc). This pipeline needs to start in Dev and be applied in all your environments, so that you know you are promoting a good base image once you get to production.

We use Kubernetes to trigger rolling restarts with new images when we release a change to the base image, and so far it's been painless, but a lot of work went into it. We use Gitlab, but any CI/CD should allow you to do it.

Re:Warp – self-contained, single binary applications by dharmab on Friday October 12, 2018 @06:25PM (#18205153)
Plop a requirements.txt file down in each repo. Use direnv or a similar tool to automatically manage virtualenvs on the local dev machines. In your container, don't use virtualenv; just add requirements.txt to the container and run pip install -r requirements.txt

See the linked code at the bottom of the page

Re:Warp – self-contained, single binary applications by dev_dull on Friday October 12, 2018 @06:34PM (#18205204)
There are so many places where this will never work. For example, if you're using openssl and you're system is missing the header files.
Re:Warp – self-contained, single binary applications by dharmab on Friday October 12, 2018 @06:44PM (#18205249)
Which is why I run and debug locally with an interactive container; installing the packages locally is just for my editor autocompletion.
Re:Warp – self-contained, single binary applications by dec0dedab0de on Friday October 12, 2018 @05:58PM (#18205033)
Have you tried pipenv? It creates a lock file that hashes all your dependencies so it makes sure you get the right one when you deploy.. And it automatically creates the virtualenv if its nog there when you install them.
Re:Warp – self-contained, single binary applications by weberc2 on Friday October 12, 2018 @07:46PM (#18205486)
We do use pipenv. I like the idea of it, but I think we ran into issues with where the pipenv lives. We used to use `--system`, but that seemed buggy and/or idiosyncratic, so we started using a venv and then we started using PIPENV_VENV_IN_PROJECT=true (for some reason) and now it's causing other issues. It seems unclear how it's supposed to be used with Docker containers, but maybe we're just overcomplicating things?
Re:Warp – self-contained, single binary applications by cjfont on Friday October 12, 2018 @05:10PM (#18204751)
JARs are okay, but they still require you to have the java runtime installed in order for them to execute.
Re:Warp – self-contained, single binary applications by pjmlp on Friday October 12, 2018 @05:39PM (#18204905)
You can bundle everything together, or use a commercial JDK with AOT compilation support.
Re:Warp – self-contained, single binary applications by dom96 on Saturday October 13, 2018 @07:07AM (#18207416)
I always wonder to what extent this is true for Go. How does it handle depending on C libraries?