Automatic Dependency Management with ElectricAccelerator

One of the well known problems with make is that it’s a real nuisance to completely specify all the dependencies in your project. For example, if you have a file main.c in your project, you probably already have a dependency like this in your makefile:

However, if main.c includes logging.h, you technically need to have a dependency like the following too — but you probably don’t have it:

The kicker is that without this additional dependency, make will fail to realize that it needs to rebuild main.o if there is a change in logging.h. Of course, this wreaks havoc on your ability to do reliable incremental builds.

makedepend and compiler-generated dependencies

The classic solution to this problem is to use one of the automatic dependency generation mechanisms, such as makedepend, or compiler-generated dependencies, if your compiler supports it (g++ does, for example, via the -MM switch). For example, you can add basic makedepend support to your makefile by adding the following lines to the end of your makefile:

Then, you can invoke make depend to generate dependencies; makedepend will append them to the end of your makefile after the magic “DO NOT DELETE” line shown above. This certainly gets the job done, but it has some drawbacks. Owen at The Grimoire has a more thorough exploration of the shortcomings, but basically the problems are:

  • It’s a manual process; if you forget to run make depend periodically, your dependencies will get stale.
  • It’s clumsy, because it relies on magic tokens in and modifies your makefile.
  • It adds complexity to your makefile.
  • It’s slow.

Some of these deficiencies can be addressed with a more sophisticated integration. My good friend Mr. Make has a great article exploring that option, as well as compiler-generated dependencies, but keep in mind that going that route will add even more complexity to your makefiles.

ElectricAcclerator autodepend

If you use ElectricAccelerator, you have another alternative: autodepend. This feature serves the same purpose as makedepend, etc, in that it automatically generates dependency information for you, so that you don’t have to manage that data yourself. The big difference is that it’s better, of course, in every conceivable way:

  • It’s truly automatic: add –emake-autodep=1 to your command-line and never worry about it again.
  • It’s transparent: the additional dependencies are tracked in a separate file, so you don’t have to modify your makefile.
  • You don’t have to change your makefiles or tool command-lines at all.
  • It’s completely toolchain, platform and language independent.
  • Best of all: it’s fast.

Autodepend uses the filesystem usage information that we already collect over the course of the build to get perfect dependency information for every file generated by the build. If the command that produces main.o reads logging.h, we’ll see that and record the implicit dependency, just as if you had added main.o: logging.h to your makefile.

Since we’re already collecting the data anyway, autodepend adds almost no overhead, but you don’t have to take my word for it. I set up a test environment that let me easily switch between makedepend, g++ -MM dependencies, and autodepend, then ran a small build using each mechanism three times. The build consisted of about 75 C++ source files, referencing about 150 header files. The times shown here (in minutes:seconds format) reflect the total time to generate dependencies and run the build itself:

Mechanism Trial 1 Trial 2 Trial 3 Average
None 3:23 3:27 3:20 3:23
makedepend 4:40 4:37 4:33 4:36
g++ -MM 4:07 3:57 3:59 4:01
g++ -MD 3:53 3:53 3:40 3:48
emake autodep 3:18 3:22 3:29 3:23

I think the results are pretty compelling: using autodepend added no appreciable overhead to the build time, while the alternatives added 20-30% to the total build time. If that’s still not enough to convince you to give it a try, maybe this will: just last week, one of our customers switched from compiler-generated dependencies to autodepend and saw their build times drop from 32 minutes to just 18 minutes.

So give it a try: you have nothing to lose — except for long builds, of course.

Update (19-DEC-2008): Added benchmark data for g++ -MD, a variant of compiler-generated dependencies that emits the dependency file at the same time the compile itself is performed, rather than as a separate step, which improves the efficiency of that technique.


Build Acceleration and Continuous Delivery

Continuous Delivery isn’t continuous if builds and tests take too long to complete. Learn more on how ElectricAccelerator speeds up builds and tests by up to 20X, improving software time to market, infrastructure utilization and developer productivity.

Follow me

Eric Melski

Eric Melski was part of the team that founded Electric Cloud and is now Chief Architect. Before Electric Cloud, he was a Software Engineer at Scriptics, Inc. and Interwoven. He holds a BS in Computer Science from the University of Wisconsin. Eric also writes about software development at http://blog.melski.net/.
Follow me

Share this:

2 responses to “Automatic Dependency Management with ElectricAccelerator”

  1. hannes says:

    Thanks for the interesting numbers. I wouldn’t have guessed that makedepend turns out to be slower than g++ -MM.

    Have you ever measured g++ -MD (i.e. dependencies generated as a side-effect of compilaton)? That should have similarly low overhead to the autodepend solution, right? It’s more complex and fragile, of course, but for those of us stuck with plain old make it seems like the best method.

    • Eric Melski says:

      Thanks for the suggestion. I reran my benchmark using g++ -MD and incorporated the results into the table above. As you can see, it is definitely faster than g++ -MM, but still noticeably slower than ElectricAccelerator’s autodepend feature (about 12% worse in this example).

      The other problem with g++ -MD, of course, is that the dependencies lag behind the current sources by one build; with the traditional g++ -MM implementation, the dependencies are regenerated first, so when you actually get to the compile phase of the build, you have good dependency information. I can imagine ways around that limitation, but they are all pretty clunky.

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe

Subscribe via RSS
Click here to subscribe to the Electric Cloud Blog via RSS

Subscribe to Blog via Email
Enter your email address to subscribe to this blog and receive notifications of new posts by email.