Andy Glew’s written a piece over on his blog opining about the need for automatic discovery of dependencies and some applications of having that data. Understanding dependencies is a requirement for any (reasonable) build tool, and all build tools allow the specification of dependencies – even shell script: specifying a serial order lets you define fully implicit dependencies! – so if you had a source for complete, accurate dependency information you could generate a makefile, or an ant build.xml, or a shell script with a serial order, from that information, thus allowing a build to use any tool.
One of the challenges to that approach in practice is that most developers have a build tool of choice (make, ant, SCons, Visual Studio, etc) and there’s a resistance to using a meta-language to describe all the other bits (compiler options, libraries, variables) which are the other necessary parts of a build system. Getting developers to write in your tool/language is the sticky wicket, since a developer is going to need to change all the little details at some point, and that developer is going to be more comfortable doing it in the tool she’s accustomed to. And some tools don’t easily map 1:1 – lots of things which work well in SCons are hard/impossible in make, and migrating make to SCons is non-trivial.
In the end, Andy’s right on, though. The dependencies are the hard, magic bit – and in large software systems with modular structures and recursive build processes, getting accurate depdendencies is *hard* – so if you can get good deps the tool platform problems can be solved in lots of ways.
Andy has a couple questions in his article I should address, since I work for Electric Cloud:
"Past conversations, either with Ousterhout or with EC sales folk,
(1) unsure as to whether they have done the truly automatic thing for dependencies
(2) sure that they are not "diversity tolerant"
- they are yet another tool that requires everyone, the whole project, to use their build tool."
1) Yep, we really have done the truly automatic thing for dependencies – ElectricAccelerator tracks at the file-system level what files are read and written, collecting the data required to build the dependency graph for a build (and also letting ElectricAccelerator know when a target was prematurely fired, in order to revert it and re-run once all implicit deps are satisfied). The approach Andy indicates earlier in his article – using strace to trap this info – is a great first pass; strace slows a build down 5x or more, though, and isn’t available on Windows. ElectricAccelerator is built for speed, so we’re more elegant about how we do the data collection.
2) ElectricAccelerator assumes as a commercial requirement that developers are tightly bound to their tools, whether by choice, edict, or (mis)fortune. A commercial product which requires code change will fail, so it’s important to us to support specific tools. So we don’t actually require everyone to use our tool – we emulate *your* tool in most cases – and compatibility with the original infrastructure is maintained.
It’s an interesting idea, writing a tool-indepdenent build language, or writing a tool which can transition from one implementation (ANT) to another (Make) – and there are commercial companies which have such products available. Electric Cloud has focused on what we see as the fundamental hard problem – seamlessly discovering dependencies – and uses it to get to what I think Andy’s end goal is:
such a meta-build might make recursive make correct.
Which is exactly where ElectricAccelerator ends up: we fix the problems identified by Miller in Recursive Make Considered Harmful by allowing all implicit dependencies to be respected (and some other neat stuff too, like merging sub-makes transparently.)
Latest posts by Electric Cloud (see all)
- Key Takeaways from Continuous Discussions (#c9d9) Podcast, Episode 85: Pipeline Analytics and Insights - March 16, 2018
- DevSecOps – It’s Not Me or You, It’s WE! - March 12, 2018
- Key Takeaways from Continuous Discussions (#c9d9) Podcast, Episode 84: Software & Pipeline Architecture for DevOps - February 16, 2018