Makefile performance: $(shell)

One rookie performance mistake I’ve seen in GNU make makefiles is the use of $(shell) without := assignment. Of course I’m not the first person to write about this, but people are still making this mistake, and it’s so easy to fix, it’s really tragic that it’s still out there.

UPDATE: read more about GNU make and the shell here!

GNU Make Variables

So what’s the problem? GNU make supports two fundamental types of variables: simple and recursive. The difference between the two is in how they are expanded. Simple variables are expanded once, when the variable is declared; recursive variables are expanded every time the variable is referenced. This is a subtle but important distinction. Consider this makefile:

If you’re not familiar with the differences in gmake variables, the output from this build may surprise you. Heck, I’m painfully familiar with the differences, and the output would still surprise me if I hadn’t read the makefile very carefully:

What’s going on here? The variable BAR_1 is declared as a simple variable, because I used := to create it. This caused gmake to immediately expand the text on the right hand side of the expression and capture the result as the value of BAR_1. At the time BAR_1 was declared, FOO had the value abc, so that’s what we got in BAR_1.

In contrast, BAR_2 is declared as a recursive variable, because I used the usual = (no colon) to declare it. In this case, gmake captures the unexpanded literal text $(FOO) as the value of BAR_2; later, when I referenced BAR_2, gmake recursively expanded the value to obtain current value of BAR_2. At that time, FOO had the value def, so that’s what we see used as the value of BAR_2.

$(shell)

$(shell) is a special function in gmake that runs an external command and captures the output for use in the makefile. For example, you could get the current working directory like this:

It is the interaction between recursive variables and $(shell) that causes the performance problem. Suppose you have a simple makefile like this:

This makefile uses $(shell) to generate a build identifier based on the date that the build was invoked. The build id is then used in the name of the output directory, to ensure that each build is written to a unique output location. Unfortunately, because I carelessly declared BUILDID as a recursive variable, gmake will invoke the shell every time the variable is referenced. Even in this tiny makefile that proves to be a surprisingly large number of references. You can see each reference with a clever little trick: add $(warning) to the declaration of BUILDID:

$(warning) won’t affect the value of BUILDID because it expands to the empty string, but it will cause gmake to print a message to stderr every time the variable is expanded. Now when we run the build, we see the following output:

BUILDID is expanded eleven times in this tiny little build, and that’s with just a few object files, and one final build product. The great thing is, it’s trivial to fix this problem. Simply by changing the declaration of BUILDID so it is a simple variable, we eliminate all but one call to the shell:

 

Tracking down offenders

If you’re using ElectricAccelerator, you can add --emake-pedantic=1 to the command-line, and emake will issue a warning for every recursive variable that contains a call to $(shell):

If you’re not using ElectricAccelerator, you can use grep to find the offenders:

So what are you waiting for? Quit dragging your build performance down with $(shell) busy work.


This article is one of several looking at different aspects of makefile performance. If you liked this article, you may enjoy the others in the series:

Build Acceleration and Continuous Delivery

Continuous Delivery isn’t continuous if builds and tests take too long to complete. Learn more on how ElectricAccelerator speeds up builds and tests by up to 20X, improving software time to market, infrastructure utilization and developer productivity.

Eric Melski

Eric Melski was part of the team that founded Electric Cloud and is now Chief Architect. Before Electric Cloud, he was a Software Engineer at Scriptics, Inc. and Interwoven. He holds a BS in Computer Science from the University of Wisconsin. Eric also writes about software development at http://blog.melski.net/.

Share this:

2 Responses to “Makefile performance: $(shell)”

  1. […] Makefile performance: $(shell): 28% of page views […]

  2. […] written before about the importance of using := assignment with $(shell). In short: not using := assignment can cause your makefile to invoke the shell far more often than […]

Leave a Reply

Subscribe via RSS
Click here to subscribe to the Electric Cloud Blog via RSS

Subscribe to Blog via Email
Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Upcoming Webinars:


Deploy Better Software, Faster
Dec 3, 11am PST
Register Now »


Continuous Discussions Online Panel

Next Episode: Jenkins in the Enterprise
Dec. 16, 10am PST


Add to your calendar »

Using CMake?
Accelerate Your Builds for Free!

Read the technical document on CMake Build Acceleration
Read Now »
banner_continuous-integration-at-scale

banner_huddle_free-build-acceleration