dib::09.16.2016+17:54
After putting it off for years at this point, I finally posted the build system (dib) that I’ve been working on since 2010 up on GitHub. It’s probably not the greatest example of Haskell code out there, since it was my first large project, but I’ve been slowly improving it over the years and I’ve tried to stay up to date with the language as much as possible. This has been an entirely free-time project, and as such, it’s only been motivated by my current needs at the time. What follows is a bunch of information on why I wrote it. Coming next time: a breakdown of the architecture.
Background
I’d been fed up with the state of build systems for years when I started the project. I liked the ubiquitiousness of Make, but the syntax, quirks, and difficulty of writing a simple Makefile to build a tree of source turned me off to it. I would use it for really simple projects, but it was a massive hassle for anything more complicated. I turned to Scons and Waf after that, but both of them were overly complicated for what I considered simple builds (it’s been a really long time since I’ve looked at them so maybe that’s changed). I did use Scons for an old Lua-based game engine I wrote, Luagame, and it was pretty successful there.
When I got a professional programming job, we used extrememly complicated Makefiles for code builds and Jam for data builds. If you’ve ever worked with Jam, recall that it has the most inane syntax and convoluted methods of building things of possibly all serious build systems. When I changed jobs, the company I went to work for was using Jam for doing code builds, and that might be one of the most complicated build setups I’ve ever seen. To put things into perspective: adding a Jamfile for a new library might only take a half hour or so; copy-paste from another library and change the directories and names in it. However, there’s a 99% chance that you made a non-obvious mistake like naming your directory with embedded upper case letters, accidentally not putting a space before a semicolon, or something even more obscure related to the way the system lumped files together into single compilation units per n library files to try to improve compilation speeds. Suffice to say, I don’t like Jam.
Goals
I finally got fed up enough that in 2010 I decided to take matters into my own hands and I laid out the groundwork for what would eventually become dib. These were the handful of high-level goals I had in mind:
- Forward, Not Backward - the majority of build systems in the wild are backwards, rule-based systems; that is, the user asks for a build product and the build system looks backward from the product to find the input files using rules that the user has set up. It continues doing this until it hits a leaf and then begins executing things from there. This also builds an implicit dependency hierarchy, which is how the build system knows how to order things. In contrast, dib is a forward build system. The user instructs it to take a set of files and do an action with them. The steps in generating an individual product (Target) are coded as a set of Stages. Each stage takes input, does an operation, and passes the result to the next stage.
- Don’t Write a Parser - a lot of build systems have made what I consider a mistake: the language used to describe the build is bespoke, often with a grammar optimized for the programmer writing the parser and not the user. Make and Jam are both guilty of this, though I accept that that decision was likely motivated by technical constraints at the time of their inception. I chose to embed dib into the Haskell language. While this has the downside of requiring a Haskell compiler, it has the upside of making the build strongly typed and giving the user access to the Haskell library ecosystem. It has also probably shaved years worth of work off the project since I didn’t have to write and then continually fix a parser.
- Try To Be Declarative - as much as possible, I tried to make the build specification declarative. That is, the user generally only needs to describe the build and not worry about the actual mechanics behind the scenes. With the exeception of writing new builders, this ends up being the case. Most other build systems choose this route, and I think it’s the only right way to do it.
- Don’t Do Extra Work - I rewrote dib twice: there was an initial prototype to prove out some concepts, and a first version that maybe wasn’t well throught-out. Part of the way through the first version, I determined that it was silly to attempt to figure out if a target was up-to-date before building it. In a forward build system, it seems to be better to just try to build the target and if nothing has changed then do nothing. That way you only evaluate timestamps/hashes/dependencies a single time versus multiple times.
- Be Straightforward and Obvious - this is something that a lot of build systems seem to fail on; once the user understands the primitives that the system offers, the mapping between them and the desired build should be obvious, regardless of the complexity of the build. I personally find forward build systems to offer this level of obviousness versus rule-based systems. In my head, at least, it follows a clearer path of logic: “what steps do I need to do to build this thing?” versus “here’s what I want, what intermediate products is it made from and what intermediate products are those made from?”
Get It
You can grab a copy of dib on GitHub. It’s MIT licensed. I haven’t uploaded it to Hackage yet, but I want to get it up there.
Next Time
In the next post I’ll be covering the system internals in much greater depth.