dib - future::10.06.2016+14:12
In the previous post, and the one before that I talked about the architecture and motivations for writing my build system, dib. In this post I’m going to go over some sticking points, bugs, and missing functionality that I’m planning on remedying in the future.
General Items
- Stop on Failure/Partial Success - Currently, dib won’t stop on failure immediately and won’t record partial successes in the database. This is pretty awful, especially for larger codebases since it will do full rebuilds of the affected Target until it builds correctly. This should be relatively easy to fix, but it’s going to be a bit of a nuanced solution.
- Reduce Database Size - The various databases are larger than they need to be right now. I’m using Data.Text as keys in a lot of places and should be using hashes instead since 32 and even 64 bits is significantly smaller than the smallest file paths will be.
- Target Validation - There are some obvious aspects of Targets that should be validated: ensuring that there’s at least one Stage and one Gatherer. Fixing this at the type level is the obvious best answer, and it looks like there’s already a library for non-empty lists. There might be other things that can be put through validation too, but I need to investigate more here.
- Dependency Caching - There’s no caching at the dep scanner level right now and it leads to a lot of repeated work, especially if one of the dependencies has a significant number of its own dependencies. I don’t have a solid architecture for how I want to handle this yet; I’m leaning toward changing the type signature of the DepScanner function to include the database, but I think it needs something stronger, probably involving the StateT monad.
- Windows Fixes - I have a local fix for this that I need to push, but currently the dib driver program doesn’t make the correct commandline to be executed by “system”, so it fails to run the actual build. There might be other issues with Windows that I don’t know about; I don’t ever test dib there.
C Builder Items
- Better C/C++ Flags Separation - There’s no separation between the flags that are used for C files and the flags used for C++ files. This should be an easy fix with minimal overhead.
- Link Flag Changes Shouldn’t Cause Full Rebuilds - Right now, if the user changes any of the options to the C Builder, it will cause a full Target rebuild. This is obviously not a good thing. I think the solution for this will be adding per-stage hashes which influence the compilation similar to how the Target hash currently does.
Possible Extensions
These are not guaranteed features, but are instead things that could end up in dib, depending largely on how much time and effort I’m willing to put in for them.
- Retrieving Output of a Target’s Last Stage in a Subsequent Target - At the moment, Targets can’t send any information from one to another. I could see this being used to inform a further Target where a library was built to, or something of that sort. However, I’m not really convinced this is a worthwhile feature, since the user can already predict where the output will end up and modify the Targets accordingly. It’s just a possibility I’ve been throwing around.
- Output Caching - Some transforms can be expensive to build: compression of textures, videos, archiving operations, etc… In a multiple user situation it would be advantageous to cache these files somewhere so other users don’t have to endure the lengthy build process if the data hasn’t changed. Being able to have a shared network location (folder, webDAV, ftp) where this cache lives and being able to pull from it would be a really useful feature. This would currently be a lot of work and would require hashing the actual file contents instead of just checking timestamps. I’d want to implement it as an optional feature that could be enabled per-Target.