Part 1: Creating a distributed system
I have recently been delving into the exciting world of distributed systems. From an architectural standpoint, distributed systems demand a different design approach than that of regular centralized applications. From a development standpoint, they provide a host of new and fun toys to play with. I have spent the last few months investigating some of the architectural styles and patterns associated with distributed systems. I have also been playing with some of the technologies used to implement these styles and patterns.
While distributed systems offer numerous benefits over centralized systems, including fault tolerance and scalability, they also have many potential pitfalls and failure scenarios. I wanted to better understand these problems and learn how to solve them. As such, I decided to set up a test lab and write some proof-of-concept code so that I could actually observe the system behavior, see how it degrades and fails, and then fix it using common distributed systems architectural styles and patterns.
In this four part series I am going to take you through one of these practical explorations and examine a common failure scenario: when one node in a distributed system doesn’t have enough resources to handle a spike in load. In part 1 I am going to go through the process of creating a distributed system that uses an RPC (Remote Procedure Call) integration style. This system will be deployed on my local box using Hyper-V to separate the nodes. In part 2 I’ll create a Visual Studio Load Test that will simulate a spike in load and bring this system to its knees. I’ll examine an alternative architecture in part 3 that uses messaging as the integration style. Finally in part 4 I’ll run exactly the same load test and see the system comfortably handle spikes in load despite having limited server resources.
Introduction
At the heart of the distributed system concept is the idea that one application needs to communicate with another either by transferring data or by requesting some action to be taken. On the surface this may seem like a relatively simple feature to implement. All we need to do is create a web service on one box and have another box call into that web service. Using this approach we can easily send data to a target application or invoke some action to be taken. What could possibly go wrong? I’m glad you asked.
The scenario
The business scenario for this exercise involves two applications. The first is an inventory application. This application is responsible for monitoring the stock levels in a warehouse and ensuring that there is always enough stock available. When the amount of available stock for an item drops too low, the inventory application sends a message to the second application, the purchasing application, asking it to submit a purchase order for the item so that the stock can be replenished.
Deployment
In order to test this scenario, I need a deployment strategy that will allow me to put the system under stress so that I can identify the failure points. While I could have simulated this distributed system by simply having the inventory application and the purchasing application run as separate processes on my local workstation, that wouldn’t have allowed me to play with the environments in which each application is deployed. Instead I decided to deploy the inventory application to my local workstation and the purchasing application to a virtual machine running in Hyper-V:
While this simulation is obviously not identical to the real thing (where the system would be deployed across different physical boxes in different locations and connected via actual network cables or wireless networks), it does allow me to play with the environment in which the purchasing application is deployed. For example, within Hyper-V there is a setting that allows me to tweak exactly how much processing power to allocate to the virtual machine. Not only can I limit the number of processors available, I can also limit the processing power of those processors. In addition to this I have the ability to limit the virtual machine’s available memory and network bandwidth.
Since I am specifically testing how my architecture holds up under stress, I want to limit the power of the environment on which it is deployed. In that regard I’ll allocate my virtual machine a single processor and 2GB of RAM. This configuration should make it relatively easy for me to push the system to its limit and see how well my architecture holds up. In a real world system I may have a lot more computing resources available but I may also have a lot more load, resulting in the same level of stress on the system.
High-Level Implementation
To implement this scenario I am going to create a new Visual Studio 2013 solution that will house the inventory application, purchasing application and test projects. Let’s take a high-level look at implementing this scenario.
The purchasing application will receive stock replenishment requests from the inventory application and create purchase orders to replenish the stock. Starting from the back end: to simulate a purchase order I’ll just write the order details to a text file. That way I can simply open up the file to see which purchase orders made it through the system successfully. If the order is in the file it means the purchasing application was successfully able to receive a stock replenishment request from the inventory application and create a purchase order. I’ll house this logic in a project called Enterprise.Purchasing.Persistence.
Next I need a way for the purchasing application to receive a request from the inventory application. One way to achieve this would be to create a web service endpoint on the purchasing application that the inventory application can call into. For that I’ll create a new ASP.NET Web API project called Enterprise.Purchasing.WebApi.
The inventory application will simply consist of a class library containing a class whose responsibility it is to send a request for stock replenishment to the purchasing application through the purchasing application’s API. I’ll call this project Enterprise.Inventory and I’ll call the stock management class StockManager.
That completes my proof-of-concept architecture. This is what my system looks like at this point:
If you are interested in seeing the full implementation of the solution take a look at this GitHub repository. It contains all the source code for this series of posts.
Implementation Details
Let’s dive into the implementation details and take a look at some code. I’m sure I don’t need to mention that the code below is grossly over simplified and does not represent best practice. I am intentionally keeping it to a bare minimum in order to ensure that the objective of this series is not lost in the details. I want to test one specific failure scenario and that is exactly what this code is designed to do.
Starting with the Enterprise.Purchasing.Persistence project: This project contains one class, PurchaseOrderStore, whose responsibility it is to persist a purchase order for an item to a text file on the local file system. This is the class I’ll use to simulate the placement of an actual purchase order. Let’s assume that this is an expensive operation. For whatever reason, it takes a long time and it is very memory and processor intensive. To simulate this, I am going to include two additional lines of code in the SubmitPurchaseOrder method. The first is both a processor and memory intensive operation: allocating a very large string. I am going to create a new string 150 million characters in size:
var bigString = new string('a', 150000000);
The second line is going to cause the executing thread to sleep for 500 milliseconds to simulate a long running operation:
Thread.Sleep(500);
So this is what my complete PurchaseOrderStore class looks like:
public class PurchaseOrderStore { private static readonly object FileLock = new object(); private const string FilePath = @"C:\Temp\PurchaseOrders.txt"; public void SubmitPurchaseOrder(string itemCode) { // Simulate a long running, processor and memory intensive operation var bigString = new string('a', 150000000); Thread.Sleep(500); lock (FileLock) { using (var file = File.AppendText(FilePath)) { file.WriteLine("Purchase order for item: " + itemCode); file.Close(); } } } }
Next we have the Enterprise.Purchasing.WebApi project that accepts stock replenishment requests from the inventory application and calls into the PurchaseOrderStore class to submit the purchase order. This is an out-of-the-box ASP.NET Web API project with very few modifications. This project contains one ApiController, called StockReplenishmentRequestsController, that exposes the stock replenishment requests endpoint. Keeping the default route, this endpoint will be accessible from [host]/api/StockReplenishmentRequests.
public class StockReplenishmentRequestsController: ApiController { private readonly PurchaseOrderStore _purchaseOrderStore; public StockReplenishmentRequestsController () { _purchaseOrderStore = new PurchaseOrderStore(); } [HttpPost] public HttpResponseMessage Post([FromBody]string itemCode) { _purchaseOrderStore.SubmitPurchaseOrder(itemCode); return new HttpResponseMessage(HttpStatusCode.Created); } }
Moving on to the inventory application, we have a class called StockManager whose responsibility it is to manage the inventory in stock. This class has a method called RequestStockReplenishment that will make a call out to our purchasing application to request stock replenishment for a particular item.
public class StockManager { private const string RequestUri = "http://myvirtualmachine/api/purchaseorders"; public void RequestStockReplenishment(string itemCode) { using (var client = new HttpClient()) { client.Timeout = TimeSpan.FromSeconds(30); HttpContent content = new StringContent("=" + itemCode); content.Headers.ContentType = new MediaTypeHeaderValue("application/x-www-form-urlencoded"); var result = client.PostAsync(RequestUri, content).Result; if (!result.IsSuccessStatusCode) { throw new Exception("Error Occurred. Status Code: {0}" + result.StatusCode); } } } }
With the inventory application running on my local machine and the purchasing application deployed to the virtual machine, we can do a simple smoke test to make sure the system is up and running. Using Fiddler I can POST an item code to the
http://[myvirtualmachine]/api/StockReplenishmentRequests
endpoint. Submitting the following request:
POST http://[myvirtualmachine]/api/StockReplenishmentRequests HTTP/1.1 Content-Length: 4 Content-Type: application/x-www-form-urlencoded =123
… creates the file C:\Temp\PurchaseOrders.txt on my virtual machine. Viewing the contents of this file shows that my purchase order was successfully submitted:
Purchase order for item: 123
With the inventory application running on my local machine and the purchasing application deployed to a Hyper-V virtual machine, my contrived enterprise distributed system is complete. In part 2 of this series, I’ll put this system under stress and see how it holds up.
You can download all the source code for this series from this GitHub repository.