Visualizing Build Processes

The other day I was waiting for my continuous integration build to finish. Think about that a minute. See anything wrong with that statement? I was waiting for a continuous integration build to finish. The whole point of continuous integration is to detect integration errors as quickly as possible. After years of growth, our builds are too long to really live up to that definition.

Obviously I have to do something about this, but what? Where do I start? The build is composed of several distinct phases: checkout, compile, unit test, package, install and system test. In addition, the complete process is actually replicated on several platforms simultaneously. In total, the process encompasses over 200 distinct steps, running in parallel on dozens of machines.

Fortunately, ElectricCommander tracks the duration of every step in the process, so I can find the “long pole” and focus my efforts there:

Unfortunately, with hundreds of steps it’s not at all practical to work with the data in this format. What I need is something that allows me to visualize the build process, so that I can see at a glance where the time is going. So I created the Plotter Plugin for ElectricCommander (click to view full-size):

Example output from the Plotter Plugin

Now instead of digging through hundreds of lines of timing information, I can see immediately where the problems are. Briefly, here’s what you’re looking at in this image:

  • Each colored box represents a single step in the build process.
  • Hovering over a box will display the name of the step.
  • Clicking on a box will load the Commander “Job Step Details” page for that step.
  • Boxes are colored according to the outcome of the step (red, yellow or green, corresponding to failure, warning or success, respectively).
  • Boxes are positioned horizontally according to the start time of the step relative to the start time of the entire process. The correlary to this is that steps that are running in parallel “line up” vertically, at least in part.
  • Steps which run subprocedures are not colored, and use a dashed outline to group the child steps.

In my case, the first problem is the two large, serialized steps near the top of the image, which correspond to execution of our system test suites on Windows. Now armed with this information, I can make intelligent decisions about how to improve my end-to-end build time.

If you want to try the Plotter Plugin, make sure you’re running ElectricCommander 3.5 or later, then download the plugin from our public plugin repository, and use ectool installPlugin to install it. If you’re not currently using ElectricCommander, you can get an evaluation copy.

Follow me

Eric Melski

Eric Melski was part of the team that founded Electric Cloud and is now Chief Architect. Before Electric Cloud, he was a Software Engineer at Scriptics, Inc. and Interwoven. He holds a BS in Computer Science from the University of Wisconsin. Eric also writes about software development at
Follow me

Share this:

3 responses to “Visualizing Build Processes”

  1. Bob Bowen says:

    Hi Eric, I can’t help but see the similarity between this and Insight. Did you use Insight technology? Will Insight eventually support Commander jobs?

    • Eric Melski says:

      @Bob: although the plotter plugin and ElectricInsight are superficially similar, it’s my feeling that they are different enough that it does not make sense to try to shoehorn them into a single app. Here’s a few examples of the mismatch between the two domains (Commander and Accelerator), which make it impractical to use a single common app for visualization and analysis:

      • Everything about Commander is web based, but Insight is a desktop app. Who wants to drop into a desktop app just for a quick visualization of a build process?
      • A significant portion of Insight is directed towards analyzing serializations and conflicts in builds, but Commander has no analogous concepts (if there’s a serialization in your Commander job, it’s because you put it there!)
      • Commander has the concept of steps-within-steps (for example, subprocedure calls and broadcast steps); Accelerator (and thus Insight) has no similar concept. You could argue that this is something like submakes, but you’d quickly find that the attributes of a step that calls a subprocedure just don’t map well to the attributes of a submake.
      • Insight is necessarily tightly coupled to Accelerator, for performance reasons. Remember, Insight is designed to handle analysis on builds with literally millions of distinct jobs. Abstracting the implementation to allow it to handle Commander concepts would hamper our ability to handle those large Accelerator builds efficiently.
      • Last but not least, the domain vocabularies don’t even match up. Accelerator has “makes” and “jobs”; Commander has “jobs” and “steps” and “procedures”. Although it is certainly possible to factor this detail out of the interface, that seems like a great deal of work to effectively make a square peg fit into a round hole.

      Thanks for the question!

  2. […] Visualizing Build Processes « The Electric Cloud Blog Share with the […]

Leave a Reply


Subscribe via RSS
Click here to subscribe to the Electric Cloud Blog via RSS

Subscribe to Blog via Email
Enter your email address to subscribe to this blog and receive notifications of new posts by email.

ElectricFlow Community Edition

free deploy automation and continuous delivery pipeline

Continuous Discussions Online Panel

Next Episode: Orchestrating Your DevOps Toolchain
February 23, 10am PST

Add to your calendar »

Dev-Test-Ops: Complex Test Automation at Intel

Insane CI Acceleration