News for hackers. Stuff that matters.
Re:SCons is not slow by moonchild on Saturday September 19, 2020 @03:48PM (#24529939)
I've been tossing around thoughts for this recently. I think the solution is something that, like scons, is built on top of a proper programming language. Unlike scons, however, it should be imperative, rather than declarative.

Core primitives, like 'a file is out of date iff its modification time or hash has changed', should be expressible 'in userspace', as it were—the core of the build system shouldn't know about them. (Though it should have some concept of a target being up-to-date and not being rebuilt, it shouldn't need to know about files.) This, combined with the ability to create proper abstractions, means you can emulate the primitives of all existing build systems and give them a common substrate.

(Bonus: assuming the core primitives are done right, you can still get performance as good as ninja. Which is good, because you won't be able to compile to a build system like ninja.)

Re:SCons is not slow by aseipp on Saturday September 19, 2020 @04:55PM (#24530262)
The answer to this is "use Shake", which is similar to what you describe. The secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files — so you can depend on environment variables, file contents, results of dynamic commands, etc — you need to be able to express "dynamic" rules where you can compute new dependencies for a rule "on the fly" while it's executing, and you need elbow grease to write it as an API. Shake also has had a lot of research done about it comparing it to other systems, is used very successfully, and has an array of nice features other systems don't, including .ninja file compatibility and build profile reports: https://shakebuild.com/

Shake is written in Haskell as a library (so your build rules have to be written in Haskell too, as programs) but you could either wire up a language via the FFI to make some classes of build rules easily expressible in JavaScript/Lua, or in theory rewrite the core of Shake in another language too. The theory is all there, though.

My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity (multi language builds, requires parsing or other "advanced" features, etc). As a bonus it's also very fast, within a few % of Ninja when parsing Ninja files, and sometimes faster than that. I was surprised when it came in as 2x as fast as Ninja on a large-ish C codebase at work, due to having picked a different "build schedule" that distributed compilations across cores better.

Re:SCons is not slow by moonchild on Saturday September 19, 2020 @07:21PM (#24530931)
Interesting. That looks like—mostly—what I was aiming for. (Though I would prefer lisp to haskell for this application, mainly because of metaprogramming.) However, I'm somewhat suspicious of this:

> A target is a file we want the build system to produce

This is somewhat arbitrary, and shouldn't be mandated by the build system. You say 'the secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files', but that doesn't seem to be corroborated?

They do show examples of 'phony' targets, but why not simply eliminate the conflation of build targets and files entirely? Phony targets are just a holdover from make.

  ______________________________________________________
The other reason I would prefer lisp (over haskell) is the lack of 'compile time'. It looks like, with shake, you have to recompile the build file every time you change it? With lisp, loading a file will be pretty much instantaneous.
Re:SCons is not slow by amelius on Saturday September 19, 2020 @05:39PM (#24530487)
> My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity

Systems evolve. You don't want to rewrite your build scripts the moment you cross the complexity line.

Your mention of Shake is nice though. However, I'm not sure if it could gain widespread adoption, since it's based on Haskell which is not very popular unfortunately (and has a bad name when it comes to learning curve).

Re:SCons is not slow by cbsmith on Saturday September 19, 2020 @06:26PM (#24530723)
Shake is interesting. I've been using bazel with a degree of success. It seems to incorporate these same concerns and scales out well to large builds. I'm not sure if shake solves enough of a new set of problems to really take hold.