Yocto Build Tutorial: BitBake Build Visualization with ElectricInsight

NOTE: Given recent updates with improved support for the Yocto Project and BitBake in ElectricAccelerator 11.0, we are revisiting our original Yocto BitBake Build Visualization post from 2013 and created the new BitBake Yocto build tutorial.  

As ecosystems such as the Yocto Project mature and become more popular for embedded device development, usage of the underlying BitBake build tool is increasing. BitBake is a powerful tool used to manage, build, and integrate complete operating system images through package and distribution management activities such as fetching of source code, configuration, cross-compilation, installation.

As with all software build systems, BitBake speed and performance are of utmost importance and a key enabler of productivity and quality. We have validated this multiple times through discussions with customers over the years.  We also validated this with key contributors and users in the Yocto and BitBake communities, where there are ongoing discussions about what can be done to improve performance.

Yocto Build Tutorial

So a valid question is, can we understand how a BitBake build currently behaves and from that, determine if, and pinpoint where, there are opportunities for improvements? The answer is a resounding YES.

In this Yocto build tutorial we will explain our process of understanding Yocto BitBake build performance and provide some useful utilities that we hope the BitBake community will benefit from as the never-ending quest for additional build performance continues!

Basic BitBake performance visualization with buildstats-plot.sh

After following the Yocto Project Quick Start guidelines and some exploration of the BitBake build process and its artifacts, we notice a folder called buildstats under ${TOPDIR}/tmp. This folder has the below structure:

buildstats/
[target architecture] (e.g. core-image-sato-qemux86)/
[timestamp]/
[packages]/
do_compile do_configure
:
:
do_unpack

An examination of one of these do_[task] files (e.g. do_compile) reveals a lot of interesting data:

>cat do_compile
Event: TaskStarted
Started: 1367381613.31
xkbcomp-1.2.4-r8.0: do_compile: Elapsed time: 3.25 seconds
CPU usage: 23.3
EndIOinProgress: 0
EndReadsComp: 0
:
:
StartTimeWrite: 1725266628
StartWTimeIO: 1725869092
StartWritesComp: 0
Status: PASSED
Ended: 1367381616.57

For each task we can get start time, end time and a bunch of other potentially useful data points. Interesting!

The BitBake and Yocto Project communities have already created some profiling and analysis utilities using this data.

The BitBake utility called buildstats-plot.sh tool generates static graphs of aggregate time spent per task-type or time spent per package. The oe-build-perf-report lets you mine performance data – and justify a lunch break or game of ping pong while it fetches data.  Interesting, yes – but not very insightful and helpful if you need to dive deeper into what’s actually going on with your build. Where are the build bottlenecks? How do we pinpoint opportunities for improvements?

Using ElectricInsight to Visualize and Understand BitBake Build Performance

ElectricInsight is a powerful tool for build optimization, troubleshooting, and analysis. Is there a way to leverage ElectricInsight and the buildstats data to visualize and understand BitBake build behavior and performance? With ElectricAccelerator 11.0, we included a conversion utility called bb2anno that transforms BitBake buildstats data into an ElectricInsight-compatible annotation file.  We can then use that data to uncover actionable takeaways such as when, and in what sequence, particular tasks are run, effects of BitBake concurrency, and overall task-by-time reporting.

Generating an ElectricInsight annotation file from BitBake build stats is done with the bb2anno utility, available as part of the ElectricAccelerator 11.0 installation (in the /opt/ecloud/i686_Linux/unsupported/ directory by default).

Following is the bb2anno command usage, where the -annoroot command is ElectricAccelerator specific and should be excluded for native BitBake visualization:

bb2anno -id <BUILDNAME> -annoroot <path_to_anno_files> -buildstats <path_to_buildstats> -buildlogs <path_to_build_logs>

The following example shows how to enter the bb2anno command to generate an ElectricInsight annotation file for a native BitBake build:

~/bb2anno -id 99937022 -buildstats tmp/buildstats/20181212102014/  -buildlogs tmp/work > combined_99937022.anno

A Real World Example

The best examples are from the real world, so we’re going to use a publicly-available Raspberry Pi Yocto BitBake build, as a reference to see what performance insights ElectricInsight can provide. In the example below we get an at-a-glance overview of the build details: total build duration, level of parallelization, and a timeline of how, when and where the individual jobs ran.

Yocto BitBake build

We can see that the build took 72m46s to complete, is using 16-way parallelization, and is composed of 3700 different jobs (i.e. BitBake tasks). Due to BitBake dependency serialization, we note there are significant sections of “whitespace” at about 12m and at about 52m. It is also easy to identify the longest tasks of the build, which suggest potential candidates for optimization.

Advanced BitBake Build Visualization with ElectricInsight

ElectricInsight has a built-in report that can be used to visualize a heat map of where your build is spending its time, called the “Job Time By Type” report. Let’s take a look at what we get from our Yocto BitBake build:

BitBake Build Visualization

As you can see, the do_compile and do_configure tasks combine for roughly 70% of the total runtime, where compile tasks average 49s and configure tasks average 42s.

This is cool! How can I use this tool to visualize and better understand my own BitBake build?

It’s really simple: ElectricInsight is bundled as part of the free ElectricAccelerator Trial Edition download.

In this blog we’ve shown how you can use ElectricInsight to facilitate performance analysis of BitBake builds, even if you don’t use ElectricAccelerator. We hope this work will be beneficial to Yocto users. If you want to try it yourself, you can get ElectricInsight and bb2anno via the ElectricAccelerator trial download.

Keep an eye out for a future blog where we’ll expand on this work by leveraging ElectricAccelerator to dramatically reduce BitBake build durations and stay tuned for next Yocto build tutorial.

Mohan Dattatreya

General Manager, Acceleration Solutions at Electric Cloud
Mohan is responsible for product management, product delivery & strategy for Acceleration solutions at ElectricCloud. Mohan brings over 20 years of leadership experience in delivering innovative products at startups as well as at large companies. Prior to joining Electric Cloud, Mohan held leadership roles at Nortel Networks, Tasman Networks, SeriousEnergy and Cisco Systems.

Mohan holds a M.S in Computer Science from Stanford University and an MBA from Santa Clara University.
Mohan Dattatreya

By continuing to browse or by dismissing this alert you agree to the storing of first- and third-party cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. See privacy policy.