This is the second of two blog posts that will share solutions to common Azure DevOps Services concerns:

Storing Service Connection Credentials

To deploy Azure infrastructure from an Azure DevOps pipeline, your pipeline agent needs permission to create resources in your subscription. These permissions are granted in Azure DevOps with a Service Connection. You have two options when creating this service connection:

  1. Service Principal Authentication
  2. Managed Identity Authentication

If you choose to use a Service Principal, you must store a secret or certificate of an Azure Active Directory (AAD) managed application in Azure DevOps. Since this secret is equivalent to a service account password, some customers may be uncomfortable with storing this in Azure DevOps. In the alternative option, Managed Identity Authentication, you configure your pipeline agent as a system-assigned managed identity and give that identity the necessary permissions to your Azure resources. In this configuration, the certificate needed to authenticate is stored on the pipeline agent within your private network.

Storing Sensitive Information in the Backlog

It is important for the User Stories in the backlog to be well defined so that everyone involved in the development process understands how to define “done”. In some environments, the requirements themselves could contain information that is not appropriate to store in Azure DevOps.

You will need to ensure your team has a strong understanding of what information should not be captured in the backlog. For example, your organization may not allow the team to describe system security vulnerabilities, but the team needs to know what the vulnerability is to fix it. In this case, the team may decide to define a process in which the backlog contains placeholder work items that point to a location that can safely store the details of the work that are considered too sensitive to store within the backlog itself. To reinforce this process, the team could create a custom work item type with a limited set of fields.

New Security Vulnerability

This is an example of how a tool alone, such as Azure DevOps, is not enough to fully adopt a DevOps. You must also address the process and people.

Storing Sensitive Information in Code

Depending on your organization and the nature of your project, the content of the source code could also become an area of concern. Most developers are aware of the techniques used to securely reference sensitive values at build or runtime so that these secrets are not visible in the source code.

A common example is using a pointer to an Azure Key Vault Secrets in your code that is used to pull the secret down at runtime. Key Vault makes it easy to keep secrets out of your code in both application development and in infrastructure as code development by using the secret reference function in Azure Resource Manager (ARM) Template or retrieving them at compile time in your Desired State Configurations (DSC).

Features in some code scanning tools, such as Security Hotspot detection in SonarQube, can be used to detect sensitive values before they’ve been accidentally checked in, but like backlog management do not rely entirely on tools to get this done. Train your team to think about security throughout the entire development lifecycle and develop a process that can detect and address mistakes early.

Storing Deployment Logs

Even if using a private pipeline agent that is running on your internal network, deployment logs are sent back to Azure DevOps. You must be aware of any logs that contain information that your organization’s policies do not allow to be stored in Azure DevOps. It is difficult to be certain of the contents of the logs as they are not typically included in any type of quality assurance or security testing, and because they will constantly evolve as your application changes.

The solution is to filter the logs before they leave your internal network. While the pipeline agent has no configuration option to do this, you can design your pipeline to execute deployment jobs within a process you control. David Laughlin describes two ways to achieve this in his post Controlling Sensitive Output from Azure Pipelines Deployment Scripts.

Conclusion

The Azure DevOps product is becoming increasingly valuable to government development teams as they adopt DevSecOps practices. You have complete control over Azure DevOps Server at the cost of additional administrative overhead. With Azure DevOps Services, Microsoft takes much of the administrative burden off your team, but you must consider your organization’s compliance requirements when deciding how the tool is being used to power your development teams. I hope the solutions I have presented in this series help your team decide which to choose.