Introducing Consul…

As the distributed systems we build continue to grow, both in number and types of services, we need better tools to discover these services, check if the services are healthy, and to provide a common and consistent configuration store. This is where “Consul” comes in: Consul is completely distributed, highly available, and scales to thousands of nodes and services across multiple datacenters.

It allows clients to always have consistent and up-to-date information of their datacenter infrastructure, which includes a datacenter-wide services and nodes discovery, health checks, and key/value storage. Internally, it uses a consensus-based election and the gossip protocol to communicate between the nodes.

Consul decouples the discovery, configuration mgmt. and health checks out of the application and facilitates them via configuration, or with very minimum code changes. Developers do not have to use the brick-and-mortar approach to build a custom solution of their own to achieve the same functionality.

In this blog post, I’ll walk you through installation and a few processes, including health checks, configuration management, and service discovery.

Installation

1) Download Consul.

2) Extract it and run the following command to run as one node server cluster that operates in development mode:

.\consul.exe agent -dev -bind="127.0.0.1"

3) Open a browser and navigate to 127.0.0.1:8500 to see the UI with the consul agent registered.

Health Checks

A health check is often a requirement to monitor applications and ensure they are available and performing as expected. Consul provides the following types of health checks that can be configured via configuration files, or programmatically from the service itself using the Consul HTTP APIs.

  1. HTTP: The consul agent triggers a HTTP GET request on a configured interval to a specified URL. The status of the service depends on the HTTP response code: Any 2xx code is considered to be healthy, a 429 Too Many Requests is a warning, and anything else is a failure.
  2.  TCP: The consul agent similar to the HTTP check triggers a TCP connection on a configured interval. The status is determined by if the service accept connections or not.
  3. TTL: Unlike the above checks, here it’s the responsibility of a service to periodically call the Consul agent to notify its status.
  4. Script/Docker: These checks depend on invoking an external application that performs the health check, and exits with an appropriate exit code. In case of docker the script will include a Docker Exec command to invoke the application running in docker.

Example of http health check definition via configuration:

1.	{
2.	  "service": {
3.	    "name": "web",
4.	    "tags": ["primary"],
5.	    "port": 8080,
6.	    "enable_tag_override": false,
7.	    "checks": [
8.	      {
9.	        "http": "http://localhost:8080",
10.	        "interval": "10s"
11.	      }
12.	    ]
13.	  }

Configuration Management

One of the fundamental challenges of modern applications, and in particular a modern distributed application, is configuration management. Consul provides a distributed fault-torrent key value store which is also reactive.

  1. CRUD operations on the key value stored can be performed via a rest API/CLI or language-specific client libraries.
  2. Keys in Consul can be organized in a hierarchy where different levels of the hierarchy are separated by the slash character (/).
  3. Reactive configuration management: Consul can “call-back” the application when a config change happens, rather than the application calling it every time for configuration changes, it facilitates this with the following options.
    1. Using long polling: Consul can “call-back” the application when a change happens, rather than have the application call it every time for configuration changes. This is achieved by using passing as a last value a custom HTTP header “X-ConsulIndex” in the request.
    2. Using Consul-Template: This is a background service that queries the consul agent or server periodically and generates/updates JSON configuration files with the latest values from the key store. These templates should be authored in GO template format.

Example of a template:

1. {
2. "DbConnection": {{ key "db/connection" }},
3. "CustomerAPI": {{ key "api/customerApi" }},
4. }

Service Discovery

With dynamic auto scaling, upgrades and failures of services, Service Discovery is a critical component in modern application architectures. Consul provides an intelligent and automatic way of discovering services that are available and healthy. The clients can query availability information via HTTP or DNS.

Consul supports both server- and client-side service discovery. With the client-side discovery, it’s up to the developers to handle the pool of services and implement logic or algorithms on which service to call. Services can be registered dynamically via code or through the configuration.

The following example registers a service running on port 8080:

1. {
2. "service": {
3. "name": "web",
4. "tags": ["primary"],
5. "port": 8080,
6. "enable_tag_override": false,
7. }
8. }

Consul Libraries and SDKs: Use this link to access the Consul API libraries and find some examples to get started with Consul.