Recently, following the awesome DevOps Enterprise Summit UK event, ElectricCloud sponsored a DevSecOps workshop with John Willis and Sonatype to give DevOps practitioners a hands-on experience with building secure continuous delivery pipelines.
TLDR: attendees built their own automated release pipelines, secured them with Sonatype security scanning, and prevented the next Equifax. It was fun hacker awesomeness – go team!
According to a Security Architect at KPMG,
“This was a great experience. Becoming a hacker for a day and building a pipeline that secures a real threat before pushing to production makes DevSecOps tangible and real.”
DevSecOps Helps Secure Enterprises
One of the challenges businesses face today is the mandate to be agile and release software faster while at the same time ensuring they’re not the next headline news for a major security breach. One of the biggest stories in recent cyber history was the Equifax breach. In 2017, Equifax was hacked and over 150 million individual’s private data was compromised. You and I were hacked and we didn’t have much choice. The sad truth is that a patch for the vulnerability that led to the Equifax breach (called CVE-2017-5638, which I’ll talk more about shortly) was available in February, but Equifax didn’t take action until nearly August. Their systems were compromised for about two and a half of those months.
Equifax lacked the proper culture and mindset to effectively prevent and remediate breaches like this. DevOps has tons to say about how to change culture to make our businesses more secure. In the context of security we call this DevSecOps. DevSecOps is just DevOps with an extra emphasis on security; it’s a way of saying – when doing DevOps, don’t forget about security!
Put in technology terms, if the goal of DevOps is Continuous Delivery, then the goal of DevSecOps is Continuous Delivery that doesn’t push vulnerable code to production.
To drive home this theme, we modeled this workshop in a way that would give practitioners a hands-on experience of securing a real cyber threat before pushing to production. To do this we built an automated release pipeline that would build a security vulnerable web-application, pull the built artifact from a Nexus repository, deploy the application into a separate environment for each stage of the pipeline, use Sonatype to run security checks on the artifact at each stage, and block deployment into production if any vulnerabilities were detected.
Let’s dive in a bit to see how this was done.
Get the Sample Web App
To start, we had each attendee fork a sample Apache Struts2 based web-application into their GitHub and configure a Jenkins server to point to their forked GitHub repo. Here’s my repository with the sample struts app, which we forked from https://github.com/Iletee/struts2-rce.
Next we had participants edit the Maven pom.xml of their sample application to point to version 2.5.10 of Apache Struts. Why? Apache Struts is a common framework used to rapidly build MVC based web applications. However, certain versions of the library contain the very security vulnerability that led to the massive Equifax security breach (called CVE-2017-5638). Version 2.5.10 is one of those versions.
So now our participants were configuring their web-applications to be built using that same security vulnerability. Cool stuff.
Create Your Pipeline
Once Jenkins was configured and the pom.xml updated we moved on to creating an automated release pipeline.
First we modeled the application in ElectricFlow and associated it with the applications’ WAR file stored in a Nexus repository (hosted on AWS). From there each participant built a pipeline from a predefined “pipeline-as-code” template, associated it with the app they just modeled, and pointed the Build pipeline stage to their Jenkins instance (also running in AWS) using the ElectricFlow Jenkins plug-in.
With this setup done, the Build stage of the pipeline would trigger a build in Jenkins (which would push the artifact to Nexus), and each following stage in the pipeline would provision an environment (also in AWS), pull the application artifact from Nexus, and deploy the app onto the target environment. At this point participants had fully working pipelines with automated build, gated stages, and automated deployments.
Secure Your Pipeline
Next participants used the Electric Cloud Sonatype plug-in to point their pipelines to a Sonatype security scanning server hosted in AWS so that each artifact would be scanned for vulnerabilities before being deployed. The most important aspect of security is early detection and getting feedback to the right people, which is why detecting security vulnerabilities in the release pipeline is extremely powerful. With Sonatype plugged in, participants now saw that when they ran their pipelines, their application failed to deploy into production because Sonatype detected critical vulnerabilities in the Struts2 WAR file, and based on this response the ElectricFlow pipeline blocked promotion to the production deploy stage.
Do Some Hacking
Now that our participants were securing the world, they wanted to play the bad guy. Since deploying a web application with a known security vulnerability to an AWS machine with a public facing IP is not what you’d call “security best practices”, we moved to the next portion of the workshop: local machines running Docker containers.
Here we had the participants pull down the WAR file they built (struts2-rest-showcase.war) to a local directory, along with the Dockerfile from the sample app in their GitHub repo, as well as a small hacker kit called exploit.py that targets the Struts2 vulnerability (a huge thanks to @Ilkka for coding this fun tool!) The Dockerfile is pretty straight forward; it just uses the application WAR file and Tomcat 7 to run a web server exposed on port 8080. We then had participants build the docker container and run it.
If performing this exercise at home, turn off your Wifi, because you’re now running a web-application with a known security vulnerability! Here’s how it looks:
Not too exciting. To give participants a visceral feel for the threat that these kinds of vulnerabilities pose we let them do some hacking of their own. The supplied ‘exploit.py’ is based off the Metasploit toolkit, and allows participants to exploit the Struts2 vulnerability. The Struts2 vulnerability we’re targeting is CVE-2017-5638, which allows injecting operating system commands into a web application via an HTTP request. With this tool, participants were able to run remote commands on their web server running in a local container, thus simulating a real attack. Here’s an example of running ‘whoami’ to see that we’re running as ‘root’ as well as viewing all the passwords used on the target web server.
Secure Your App
After having fun playing unethical-hacker, we used the Sontatype report to get a detailed look at the vulnerabilities present in the various versions of the Struts2 library (struts2-core) and found that version 2.5.13 was the first version released without any security vulnerabilities.
The final step in the workshop was to have participants update their app’s pom.xml to reference Struts2 version 2.5.13, and re-run their pipeline. Doing so yielded clean scans at all stages of the pipeline, and a successful deploy to production.
This workshop was a great opportunity for DevOps practitioners to get hands-on experience with a real world security vulnerability. They saw first-hand how critical it is for the technologies that businesses use to design, build, and release software to effectively detect, notify, and prevent vulnerabilities. The good news is, with the proper security tools and a robust continuous delivery process in place, we can look forward to a world with less Equifaxes.