Before you start deep dive for implementing DevSecOps in this blog post, please review the fundamentals of DevSecOps in my first blog post. It will help understand the ‘Sec’ in DevSecOps and get up to speed on various security tools for implementing DevSecOps in your CI/CD pipeline. Although there are many code repositories tools with CI/CD built-in, this blog walks through GitHub and its security scanning tools for DevSecOps implementations.
This blog post provides a GitHub repo for you to fork and try it on your own. The repo has an out of the box .NET core application, docker file to containerize the application. GitHub actions workflow YML code to build and deploy the containerized application to Azure. Further steps below will help understand introducing code scanning and container scanning tools. While I have used CodeQL for code scanning and Anchore for container scanning, can easily be replaced with other security scanning tools.
This repo/blog does not have all DevSecOps pipeline security features but integrates code scanning and container scanning tools. It shows how to get started on a GitHub Actions workflow and add these tools for security scanning. DevSecOps fundamentals are to understand and integrate these security tools in your pipeline. Source code is the single source of truth, and adding CI/CD pipeline automation is the first step. The security tools integrated into the pipeline scan your source code, scan the dependency packages/software, and will be added to your code.
GitHub Actions CI/CD
As we saw in the previous blog post, we will use the below high-level DevSecOps CI/CD pipeline workflow. I am not covering ‘artifactory’ functionality in this blog post and covering it in the next blog post!
Pic: Simple DevSecOps pipeline workflow.
A .NET core application is then added to the GitHub repo. The docker file is added to containerize the .NET Core application using docker build. A GitHub actions workflow is added to run the CI/CD pipeline. The actions will contain code for build/deploy and integrating security tools within the pipeline. The GitHub Actions workflow in the repo has the following steps:
- Starts the workflow manually which helps with debugging/testing. It can be changed to run when a branch is pushed or to main, and when a Pull Request is submitted.
- Azure related variables are declared as workflow environment variables
- Build the .NET code
- Parallelly CodeQL scans the code. CodeQL action can be added to the main pipeline YML.
- Anchore container scanning before the container deployed to ACR
- Log in to Azure
- Create an Azure Container Registry (ACR) and log into ACR.
- Docker build and push the application container to ACR
- GitHub container scan, scans the container in ACR
- Create a new Azure Container Instance to host the application running in a container
- Log out from Azure after completing the pipeline workflow
Now log into the Application is successfully running in Azure. Log in & verify your application running.
To learn more about GitHub Actions workflow and how to leverage your projects, follow the links:
CodeQL
Code scanning is essential even before security scanning is applied in the pipeline, as this is the first defense line. GitHub blog, GitHub code scanning is a developer-first, GitHub-native approach to find security vulnerabilities before they reach production easily. We’re thrilled to announce the general availability of code scanning. You can enable it on your public repository today!
The code scanning is completed using CodeQL, which is GitHub native approach to scan the code to identify the security vulnerabilities while the code is still being built. Other third-party code scanning workflows can also be added, as you see from the below screenshots. To set CodeQL code scanning, click on ‘Security’ from your repo. Then select ‘Code Scanning’ to add CodeQL workflow.
Pic: Add Code Scanning to your code in GitHub repo.
Pic: Add CodeQL workflow to your code. Third-party code scanning can be added as well.
Pic: CodeQL Code scanning results.
CodeQL code scanning workflow runs parallel to the main workflow. It can be changed to run at the beginning of the main workflow also. CodeQL identifies what type of language is in the repo and runs the scanning accordingly. As this repo contains C# and JavaScript, code scanning is done for those languages. If there are significant errors/vulnerabilities found, then the code scanning workflow will fail. But for general warnings, alerts are available for review and fix the warnings.
Pic: View the code scanning alerts.
Pic: Code scanning results.
CodeQL scanning warnings can be analyzed and decide whether it is needed to fix the errors/warning and if it is safe to ignore. As this is the out of box .NET application, there are not many alerts/warnings. But if its production code and especially any migration-related codes might have more security vulnerabilities that need a fix before proceeding in the CI pipeline.
GitHub Container Scanning
This container scanning is native GitHub action to scan Docker containers in the CI pipeline. This identifies any vulnerabilities before the container is published to an application instance. More information can be found here, and this action uses Trivy and Dockle for running container scans on these images. They follow CIS Container Benchmarks for a baseline to secure the container images.
Pic: GitHub Container Scan is added to the main workflow.
Having this action in the repo and running the main workflow failed due to these vulnerabilities findings. Any of the vulnerability findings can be ignored if it is in acceptable criteria. They can be added to ‘allowedlist.yaml’ file. The container action will then ignore those vulnerabilities. For more information, read here. Container scanning in GitHub is robust in scanning for vulnerabilities before the containers are deployed to the customer environment. But a closer review of these vulnerabilities needs to be done so appropriate scanning is performed in the repo. An analysis will yield better evaluations to find if the findings are indeed from code or from any dependencies. The container scan provides scantizer results like this.
Pic: Vulnerability findings.
Pic: CIS Container baseline violations warning & info.
Pic: Added vulnerabilities to ignore list.
Anchore Container Scanning
Anchore is an open-source container scanning tool added to the GitHub Actions pipeline. More than one container scanning actions can be added to a repo workflow—more information on how Anchore container scanning works.
Pic: Anchore container scanning action added to main pipeline workflow.
Pic: Anchore container scan also identified the same vulnerabilities as GitHub container scan.
According to NVD (National Vulnerability Database) in NIST, CVE (Common Vulnerabilities Exposures)
CVE defines vulnerability as:
“A weakness in the computational logic (e.g., code) found in software and hardware components that, when exploited, results in a negative impact to confidentiality, integrity, or availability. Mitigation of the vulnerabilities in this context typically involves coding changes, but could also include specification changes or even specification deprecations (e.g., removal of affected protocols or functionality in their entirety).”
All vulnerabilities in the NVD have been assigned a CVE identifier and thus, abide by this definition.
I took one of the CVE identified in both GitHub & Anchore container scan actions, CVE-2019-3843. According to NVD database in NIST, this is related to systems service-specific running in the container instances. As this repo is out of the box .NET core application, no additional code has been added. Further analysis must be done to understand the CVE depth to mitigate or override.
Pic: https://nvd.nist.gov/vuln/detail/CVE-2019-3843#vulnCurrentDescriptionTitle
The vulnerabilities could be coming from the dependency container image used for building the docker image for the application. Hence it is essential to make sure dependencies are scanned as well.
Application Container Deployed in Azure
If all scanning goes well, then the pipeline is ready to deploy the application to Azure. Here the Azure CLI is used in the pipeline to deploy the containerized application to Azure Container Instance (ACI)!
Pic: Resource group for Azure resources.
Pic: Azure resources (ACR & ACI).
Pic: Container in ACR.
Pic: Container deployed to Azure Container Instance (ACI)
Pic: Application up and running in Azure.
I hope the DevSecOps implementation walk-through with GitHub repo to try it out on your own adds value in understanding ‘Sec’ in DevSecOps and how to protect the codebase from vulnerabilities before deployed into production. While the blog post walked through GitHub CI/CD, all the other DevOps tools such as Azure DevOps, GitLab, Jenkins, etc. provide similar security tools implementations in their CI/CD pipeline. So, make sure to integrate security tools and shift the security to the left in whichever DevOps tool your organization decides to implement for your CI/CD pipelines. Follow up back here for the next blog post in this series for artifactory, running security tools from containers, the importance of containers, Kubernetes, and how to consume hardened container images from IronBank offered by DoD DevSecOps!
