How Sharing Really Is Caring – with ElectricAccelerator Shared JobCache

sharing-is-caring

This is a follow on to our post about Android bottlenecks, where we talked about how ElectricAccelerator can shorten Android builds by 300%.  A few weeks ago we introduced v 9.0 of ElectricAccelerator that included enhancements to the JobCache add-on, including the Shared JobCache feature.  We’d like to take a few minutes to illustrate how this works, and how your development efforts can benefit from it.

First, let’s review a bit about JobCache and how it works.  JobCache is an optional feature of ElectricAccelerator that intelligently analyzes past compilation tasks along with recent changes, and then smartly reuses object files that aren’t impacted by any of the recent changes.  In other words, it shortens build times by figuring out what parts of the build don’t have to be recompiled.

Caching is nothing new.  ccache has been around for years but JobCache has some significant advantages:

  • It’s up to 300% faster than ccache, especially on Android builds.
  • It’s designed for reliability so that it can even be used for production builds – a well-known risk for ccache.
  • The visualization tools make it easier to identify and correct build bottlenecks – a near-impossibility with ccache.
  • It’s more collaborative by spreading the benefits of eliminating costly and redundant efforts of rebuilding unchanged modules across the whole team.

We recently attended Agile 2016, DevOps Enterprise Summit, and AppSphere 2016 and surveyed nearly 1,000 attendees about their development and deployment challenges.  We’ll discuss the results of those surveys in depth at another time, but one statistic that jumped out at us was that over 50% of respondents complained that they frequently experienced problems because of manual processes, workflow and dependency complexities, environmental differences, and deployment failures.

release-challenges-ea

Electric Cloud in-booth survey of 1000 attendees from Agile 2016, DOES 2016 and AppSphere 2016

While we didn’t survey for the root causes of each of those types of issues, people told us one common source of those problems: developers hadn’t used the latest build components when they did their builds.  They were relying on the contents of their own output trees and local caches without checking to see if newer versions were available.

Ouch.  That’s not helpful if your bonus is tied to timeliness or quality metrics.

Not only does Shared JobCache eliminate the drag of rebuilding components that were unchanged, it eliminates the more costly problem of reusing the wrong or outdated components, too.

So how does Shared JobCache work?

Whether you are using JobCache locally or shared, the basics are that it maintains a slot for each combination of command line options, relevant environment variable assignments, and current working directory. A slot can be empty or can hold a previously-cached result. If the appropriate slot holds an up-to-date result, a cache “hit” occurs, and compilation is avoided.  A cached result becomes obsolete if eMake detects that the contents of any of the files needed for that compile have changed in addition to checking all of the combinations mentioned above. Such file changes include any files that are read during compilation, which means all source files, precompiled headers, and compilation tools included in the eMake root. JobCache is able to detect file system changes by using the distributed file system called Electric File System that is part of ElectricAccelerator.

To get going with Shared JobCache someone from the Release Management Team designates the shared cache in a path name.  Individual developers get a local instance of JobCache by default.  When a “golden” build is run, overnight, for instance, the results are also copied to the Shared JobCache.  When an individual developer does a build of their morning’s work, they reference the Shared JobCache path in their own make files.  That build automatically draws unchanged objects from the shared cache, ensuring the entire team is working from the latest version of everything.  If a developer changes one of those components, the MD5 hash will be different and emake recompiles that changed component during the build.

shared-job-cache

Golden Output results are copied to the Shared JobCache so individual developers can automatically reuse unchanged components

Users of IBM’s ClearCase will recognize that Shared JobCache feature provides the same functionality as the winkin feature in ClearCase.

Your Parents were right: Sharing makes everything better

It’s bad enough wasting the time to rebuild a component that didn’t need to be rebuilt.  It’s even worse, and potentially more costly, to take the time to rebuild something because you or someone from the team didn’t use the latest version of a component.  ElectricAccelerator avoids both of those scenarios and can provide up to 300% speed improvements over open source tools like ccache.

In an up-coming post, we will show how the visualization and build analytics tools within ElectricAccelerator go beyond dependency management to actually predict and control build times based on the number of CPUs in the cluster.


banner_huddle_free-build-accelerationAccelerate your Android Builds for free

Learn More about ElectricAccelerator and Download ElectricAccelerator Huddle Now!

 

Mohan Dattatreya

Mohan Dattatreya

General Manager, Acceleration Solutions at Electric Cloud
Mohan is responsible for product management, product delivery & strategy for Acceleration solutions at ElectricCloud. Mohan brings over 20 years of leadership experience in delivering innovative products at startups as well as at large companies. Prior to joining Electric Cloud, Mohan held leadership roles at Nortel Networks, Tasman Networks, SeriousEnergy and Cisco Systems.

Mohan holds a M.S in Computer Science from Stanford University and an MBA from Santa Clara University.
Mohan Dattatreya

Share this:

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe

Subscribe via RSS
Click here to subscribe to the Electric Cloud Blog via RSS

Subscribe to Blog via Email
Enter your email address to subscribe to this blog and receive notifications of new posts by email.