Agility and Discipline are Both Required as Organizations Adopt the Cloud

At the beginning of cloud adoption, experimentation is common and exploration of applicable technologies is welcomed. Development teams are excited to try the newest service offerings and proof of concept efforts are completed with great interest. However, the thrill of discovery often turns to frustration as configuration complexity increases, and cost control challenges mount. The initial exploration proof of concepts often does not consider where costs are coming from or what resources are assigned to different teams.

While cloud providers make it easy to provision new resources, considerable forethought is necessary to come up with disciplined procedures to measure costs and control access. Security and management raise valid concerns regarding proper security controls. Infrastructure as Code (IaC) concepts will allow for exploration and development agility while including the proper management review, thoughtful allocation of resources, and cost control initiatives. In this post, we will introduce IaC concepts and tools that can be used to bring agile interaction to your cloud adoption experience.

What is Infrastructure as Code (IaC)?

Infrastructure as Code (IaC) refers to the tools and processes used to track and manage infrastructure resources as code. This code is commonly stored in a version control system and the tools are most effective when applied to virtual environments. One such example is the Microsoft Azure public cloud, but IaC can be equally effective in private cloud environments. The combination of version control with executable infrastructure definitions provide a complete change history and ensure future repeatability

Historical Configuration Management

IT administration teams have historically consisted of a handful of individuals tasked with managing a large footprint of physical and virtual machines. With the popularization of virtualization technologies came the need to effectively maintain configuration and security baselines. Rudimentary automation tools such as parallel SSH gave way to more powerful configuration management tools such as Puppet (2005) and Chef (2009). These tools allowed a single IT administrator to establish and apply configuration templates across thousands of machines, and address divergence from configuration when needed.

With the maturation of the configuration management tooling, different paradigms began to emerge. Puppet and Chef both operated with a pull model, utilizing a machine agent responsible for retrieving configuration from the central control server. Newer entrants to the Infrastructure as Code space such as Ansible (2012) and Terraform (2014) operated with a push model over established protocols (such as SSH) for remote access. Configuration changes also saw two very different approaches to how they were defined between Infrastructure as Code tools. The imperative model relied on the developer to define specific change operations, while the declarative model favored the definition of the desired end-state. In the declarative model, the IaC tool itself was responsible for identifying and applying changes. Each model comes with unique benefits and challenges, but regardless of the model chosen, the application of IaC tools has become a huge force multiplier for the IT team.

DevOps

In the last few years, the term DevOps has emerged out of the convergence of development and operational teams adopting shared tooling. This new engineering specialization brings with it a mindset founded in the security and configuration management needs of the IT administrator while enhancing it with version control and module abstraction common in the software development space. Probably the biggest change brought forth by DevOps was the adoption of container technologies, largely popularized by Docker. Containers greatly hastened the adoption of immutable infrastructure. The underlying machines became less important themselves, instead, acting as a vehicle to deploy container workloads.

Immutability in Infrastructure

Infrastructure immutability refers to the static nature of the machines themselves and is most easily achieved in virtualized environments such as Microsoft Azure. The value of immutability is that infrastructure configuration can be reasoned about based on a single source of truth – the code that resides in the version control system. Configuration drift in long-lived infrastructure systems is a monumental challenge that Puppet and Chef have worked to solve, but this problem can be eliminated entirely by treating our infrastructure as replaceable. When changes are required, we simply replace old infrastructure with new ones.

Adopting immutability in infrastructure configuration is not an overnight process. There are considerable challenges, especially when dealing with stateful systems such as databases or file storage systems. While these challenges are real, the advantages are substantial:

  • Deterministic infrastructure
  • Version controlled source of truth
  • Infrastructure automation via CI/CD systems
  • Audit change deltas prior to execution
  • Peer Review based on code change and change deltas
  • Gated approvals from security authorities

Adopting IaC Within Your Organization

Adopting Infrastructure as Code in your organization must be approached carefully. You may be met with resistance from security and IT staff. Successful adaptation requires your stakeholders to understand the benefits of the IaC method.

Greenfield cloud projects are one of the best ways to introduce these concepts without having to tackle the challenging task of codifying existing systems. This approach has the advantage of a reduced scope while learning the tools and exploration without danger of impacting operational systems. If a greenfield project is not possible, this can be done with a short-run proof of concept effort. In the early stages of a project, attention to the following guidelines will help set you up for future success:

  • Security first
  • Avoid premature optimization
  • Favor managed (PaaS/SaaS) vs unmanaged (IaaS)
  • Tag resources aggressively

Security First

Security first is one of the most challenging things to get right in any project, but cloud efforts are especially challenging as there is a much higher exposed surface area. With consuming data from object storage APIs you could expose sensitive data to the world if not properly locked down. There are known security breaches every year where misconfiguration caused huge leaks of Intellectual Property and Personally Identifiable Information. Infrastructure as Code is uniquely suited to minimize the pain associated with good security practices. Least privilege is a key tenant of security and Infrastructure as Code can be used to define access policies that explicitly allow only the resources that need them. This is made simple because resource identifiers can be dynamically injected into policies as the Infrastructure as Code is responsible for managing both.

Avoid Premature Optimization

Premature optimization of systems or code is a trap that can be easily fallen into. Infrastructure as Code offers a wealth of opportunity to create reusable modules for resources. These modules bring advantages for sharing proven code between environments, but if used too early can greatly slow down team velocity. It is generally best to iterate your system until you reach a secure, stable MVP where the architecture is well established before beginning the abstraction into discrete modules.

Favor managed Paas/Saas

Deciding between Platform as a Service (PaaS), Software as a Service (SaaS) and Infrastructure as a Service (IaaS) is straight-forward early in a project. SaaS offerings, such as Azure Databases, are generally always the right choice over IaaS with a self-managed database. SaaS or PaaS is able to eliminate the busywork of security patching and backup management, allowing the team to focus on using the software to solve their unique business challenges. Time is our most valuable resource and while IaaS has its place, it incurs the highest time expenditure to manage properly.

Tag Resources Aggressively

Resource tagging is fundamental to cost visibility in cloud environments. It is difficult to devise a complete resource tagging strategy, but at a bare minimum, all resources should include a project tag. Even if your project is the only one present in your cloud account, allowing tags to be set on all resources by your Infrastructure as Code will make it easy to adopt uniform tagging in the future. Tags can also be useful in differentiating development, test, staging, and operational environments if they reside in a shared account.

Summary

This is just the first post in a blog series on DevOps and Infrastructure as Code. Understanding the concepts outlined here is the foundation of successful cloud adoption. In the future, we will look at applying these principles with specific examples that can be used as a template in your own DevOps journey. In the meantime, some resources can be found below that expand on topics discussed within this post: