News for hackers. Stuff that matters.
Re:Why Use Make (2013) by barbazoo on Wednesday January 11, 2023 @02:15PM (#34343084)
> Note: use tabs rather than spaces to indent the commands in your makefile. Otherwise Make will crash with a cryptic error

Using this as an example, are there more modern equivalent tools that may be a bit more user friendly? I appreciate make and I get its age and the complexity and all, it's just sometimes I need something that's simple and explicit, without the historic baggage.

Re:Why Use Make (2013) by chubot on Wednesday January 11, 2023 @02:34PM (#34343366)
Another commenter and I both wrote some simple wrappers around Ninja in Ruby and Python -- I would recommend that approach

https://news.ycombinator.com/item?id=32303692

I started from ninja_syntax.py and then improved it based on what I needed. Ninja is fast and simple, but it's (intentionally) lower level than you might want.

https://www.oilshell.org/blog/2022/10/garbage-collector.html...

It does seem like there should be a "standard" thing that is maybe not so C++ specific (like CMake and Meson which both seem to use Ninja), i.e. for the use cases in the blog.

Re:Why Use Make (2013) by drothlis on Wednesday January 11, 2023 @05:12PM (#34345766)
I love Python + Ninja! I was going to link to my article on Ninja + OSTree, but I found in your blog you've already seen it. :-)
Re:Why Use Make (2013) by chubot on Thursday January 12, 2023 @12:00AM (#34349412)
Oh cool, well it's worth linking here!

https://lwn.net/Articles/821372/#objects

I linked it from https://www.oilshell.org/blog/2021/04/build-ci-comments.html...

How is OSTree working out? Are you still using it?

It seems cool, but for some reason I don't see it mentioned very much, maybe because it's focused more on embedded system image uses, rather than "the cloud" which seems to be more popular on this site?

I would like to see Docker/OCI containers "broken up" into just file formats and just content-addressed data/networking. Not a weird registry with a different local API than remote API. It should be more like git !

Not sure if OSTree fits there -- as far as I remember, it's inspired by git but it's focused on a different use case. This recent project seemed interesting but it's also for more of a machine learning use case: https://news.ycombinator.com/item?id=33969908

Re:Why Use Make (2013) by drothlis on Thursday January 12, 2023 @06:33AM (#34352062)
Fixed your link to the full article: https://lwn.net/Articles/821367/

OSTree is still working very well for us. At the time I wrote the article we had been using OSTree (and the build system I described in the article) for 4 years; 3 years later not much has changed. In those 3 years we have started building images for different architectures and our build system / OSTree handled it just fine, as you'd expect (it already handled cross-compilation, but for a single architecture).

Our build system also builds Docker containers (for our cloud services) but instead of a Dockerfile we use a custom yaml format that lists the base image (like "FROM" in a Dockerfile), the apt packages to install, and the commands to run (like "RUN"). Then we create a lockfile of all the apt packages, download them (into a local OSTree repository, but that's an implementation detail), and install them with a custom "docker run" command + "docker commit". We end up with the base layer + the apt layer + a single layer produced by concatenating all the RUN commands + a layer with our compiled Rust binaries and Python files. We use apt2ostree to generate the lockfile (really it's our patched version of aptly doing the work) but we use docker (not OSTree) to build the Docker layers. We use Docker's standard push/pull mechanisms to deploy these containers.

To hook this up to Ninja we use "marker files" (a file in the build directory) to track whether this work has been done (e.g. you need to regenerate the apt layer if that layer's "marker file" is older than the lockfile).

Our Python "configure" script (i.e. our build system) looks something like this:

    def build_docker_container(name):
        yamlfile = f"{name}/docker-base-image.yaml"
        with copen(yamlfile) as f:
            data = yaml.safe_load(f)
        ubuntu_version = data["ubuntu_version"]
        image = docker_pull("ubuntu:{ubuntu_version}")
        image = docker_apt_install(image, data.get("apt_dependencies"),
                                   lockfile="%s/Packages-%s-amd64.lock" % (
                                        name, ubuntu_version),
                                   ubuntu_version=ubuntu_version)
        cmd = "..."  # RUN commands from yaml
        deps = [...]  # dependencies from files listed in yaml
        return docker_mod(image, cmd, deps)
(Where "copen" is like "open" but it tracks which files have been read by "configure" itself, to detect if we need to re-run "configure").

What I really like about the Python+Ninja combo is that you can pass these targets around as Python variables — you don't have to come up with an explicit filename for each one (the target name is generated by each helper function, e.g. `docker_pull` returns "_build/docker/${name}"). This makes it so convenient to compose these build rules. And if you ever need to debug your build system, everything is very explicit in the generated ninja file.

We don't have a ton of these containers so it has worked well enough for us because the apt layer changes rarely, and the layer above it is small/fast. The main thing we get (that you don't get from vanilla docker/apt) is lockfiles and reproducibility.

Re:Why Use Make (2013) by drothlis on Thursday January 12, 2023 @07:56AM (#34352789)
I realised this is another example of "trees" as first-class citizens in a build system. In my comment above the tree we're passing around is a docker layer; in my LWN article it's an OSTree ref. We use the former for our cloud containers, the latter for our embedded device rootfs and systemd-nspawn containers.

I suppose we could use systemd-nspawn on our cloud servers too, instead of docker, but when we wrote the build system we were already using docker so it was the expedient thing to do at the time.

Re:Why Use Make (2013) by chubot on Friday January 13, 2023 @12:55AM (#34364419)
Hm yes now I remember that point about how the file system targets are anonymous Python objects that you can pass around to functions.

Are there any open source examples? I looked around the github account, but I mostly remember this tool

https://github.com/stb-tester/apt2ostree

I'd be interested in seeing the Python config and Ninja output, to see how it works. Right now it looks to me like the dependencies are more implicit than explicit, e.g. with your copen example

---

The system I ended up with is more like Bazel, but it's not building containers, so it's a slightly different problem (although I guess Bazel can do that now too). But I'm interested in building containers incrementally without 'docker build'.

I made some notes on #containers in the oilshell Zulip about incorrect incremental builds with docker build. It's also slow and has a bunch of ad hoc mechanisms to speed it up

I like the apt lockfile idea definitely ... However I also have a bunch of other source tarballs and testdata files, that I might not want to check into git. How do you handle those -- put them in OSTree?

Our config looks like this

https://github.com/oilshell/oil/blob/master/core/NINJA_subgr...

and there are other /NINJA_subgraph.py files. The library they all use is in build/ninja* of the same repo. Thanks for any details!

Re:Why Use Make (2013) by Akronymus on Wednesday January 11, 2023 @02:24PM (#34343221)
I'd say Shake fits your needs. https://hackage.haskell.org/package/shake
Re:Why Use Make (2013) by imran-iq on Wednesday January 11, 2023 @02:26PM (#34343251)
Since make 3.82 (which is about 12 years old now) supports `.RECIPEPREFIX`

  .RECIPEPREFIX = >

  all:
  > foo bar
Re:Why Use Make (2013) by enriquto on Wednesday January 11, 2023 @05:34PM (#34346021)
also, since forever, you can use a semicolon and put the rule on the same line:

    target : deps ; rule
There are certainly a lot of legitimate criticisms of make, but the tabs issue is ridiculous (as are complains about .PHONY).
Re:Why Use Make (2013) by kortex on Wednesday January 11, 2023 @07:44PM (#34347438)
Tabs are absolutely a valid criticism. I can't tell you how many times the editor I'm using put in 4 spaces instead of a proper tab.

.PHONY is truly a bizarre construction. There's nothing like it in modern code systems. It's only by familiarity with make that it gains some semblance of normalcy.

Re:Why Use Make (2013) by dxhdr on Wednesday January 11, 2023 @02:25PM (#34343239)
> Using this as an example, are there more modern equivalent tools that may be a bit more user friendly? I appreciate make and I get its age and the complexity and all, it's just sometimes I need something that's simple and explicit, without the historic baggage.

Indenting with tabs too complicated? Let me introduce you to YAML.

Re:Why Use Make (2013) by barbazoo on Wednesday January 11, 2023 @02:27PM (#34343277)
> Indenting with tabs too complicated?

I meant "Otherwise Make will crash with a cryptic error" and used that as a single example.

Re:Why Use Make (2013) by hedora on Wednesday January 11, 2023 @04:38PM (#34345295)
I've found the "make tabs are confusing" crowd heavily overlaps the "python is more readable than perl" crowd.

I find the cognitive dissonance amusing.

Re:Why Use Make (2013) by lmm on Wednesday January 11, 2023 @05:14PM (#34345792)
Both views come from the same place: the idea that the semantic scoping should match the visual scoping, rather than counting tabs for indentation but not counting visually indistinguishable spaces (make) or ignoring your indentation entirely in favour of what the braces said (perl).
Re:Why Use Make (2013) by int_19h on Wednesday January 11, 2023 @06:36PM (#34346690)
YAML prohibits tabs outright. That sounds a lot simpler to me, since there's no need to be aware of the difference between a tab and the equivalent number of spaces when the former is not a possibility at all.
Re:Why Use Make (2013) by hawski on Wednesday January 11, 2023 @02:28PM (#34343284)
I don't understand why and don't know when spaces were started to be used for indentation. It never made sense for me. Your code editor surely supports setting tab width if you are not happy with the default.

If the answer is vertical alignement I will weep and offer pity, but no sympathy. Even if that is the case you can still indent with tabs and aligns with spaces if you like to torture yourself, others and your RCS' history.

Re:Why Use Make (2013) by lanstin on Wednesday January 11, 2023 @03:55PM (#34344622)
I have preferred spaces to tabs since the mid nineties because everyone sees it the same. With tabs people use 2 4 and 8, which can make my tendency to over indent code cause more trouble.

I will say I have been doing go recently and have been forced into tabs and camelCase, and have found consistency really is better than The Right Way.

For make, since using protobufs and code gen and being allergic to checking generated code in, have found make to be nicer than I remember it.

Re:Why Use Make (2013) by burnished on Thursday January 12, 2023 @12:25PM (#34357211)
Im no historian but I suspect white space sensitive languages influenced the desire to drop a white space character.