Tech Tip

Setting Up Blue-Green Deployments in IBM UrbanCode Deploy 6.2.4

Print

Advanced deployment strategies such as blue-green deployments and rolling deployments are critical for managing multinode installations of applications that must be updated without interruption. Blue-green deployments fit these requirements because they provide smooth transitions between versions, zero-downtime deployments, and quick rollback to a known working version.

In IBM® UrbanCode™ Deploy version 6.2.4, we’ve added features that make complex deployments like these easier to set up and manage. We’ve enhanced features that let you update a few nodes at a time and verify that those nodes are working before you commit to a new version. The result is more control and better usability when updating environments with a large number of systems.

Most of these enhancements are to loops within application processes:

  • We’ve enhanced the For Each Agent loop to let you control which agents run automated tasks. This way, even if you have a large number of agents, you can run automated tasks on specific groups of agents by tagging agents and running tasks only on agents with one or more of those tags. For more information about this step, see For Each Agent.
  • We’ve also added the For Each Tag loop, which is similar; it runs once for each of the resource tags that you specify. So you can organize the resources in your environment by tag, and then set which tags to run on. You can also set the order of the tags in this loop, so for example, you can deploy to canary nodes first and all remaining nodes second, all in the same process. For more information about this step, see For Each Tag.

With these steps, you have great flexibility in deploying to complex and multi-node environments.

In this article, I’ll cover how to set up a simple blue-green deployment using these new features. For a video demo of this setup, see Blue-green deployments with IBM UrbanCode Deploy.

The blue-green deployment, step-by-step

Here’s a quick summary of blue-green deployments. There’s lots of other information out there if you want more detail.

A blue-green deployment uses two identical production environments, referred to arbitrarily as Blue and Green. (Note: that does not mean actually duplicating the production hardware, as I’ll explain later.) My end-users are always using one environment, and the other environment is inactive and available to host an upgraded version. When the inactive environment is working fine with a new version, I set the load balancer to send traffic to the new version. If anything goes wrong, I can quickly switch back to the old version, just by flipping the load balancer back. If all is well on the new version, I start the cycle again when another version is available, by installing that new version to the newly inactive environment.

The blue-green environment, represented physically

Of course, it would be wasteful to have two copies of the production environment. It’s better to have two environments that are logically separate, but are running on the same hardware. The separate logical environments benefit from running on identical physical environments, so I can be sure that the two environments behave in the same way and have the same dependencies.

There are many ways to maintain separate logical environments like this, but I’ll cover just one straightforward method here: running the Blue and Green environments on different ports on the same servers. The Green set of resources runs applications on one port, and the Blue resources run applications on a different port. That way, when it’s time to switch from one environment to another, I just change the load balancer to direct traffic to a different port.

A diagram of a blue/green deployment with three Blue nodes and three Green nodes

The blue-green environment, with nodes hosting both the blue and green environments

Here’s a topology diagram that shows how the systems might be set up. In this case, each node hosts two instances of Apache Tomcat, each running on a separate port. Each instance of Tomcat hosts a copy of the application. You could also set up the applications and application servers in other ways, such as setting up a single instance of Tomcat and deploying the applications to different context roots.

A topology diagram of a system with blue and green application servers that are running on overlapping nodes

Example scenario

Here’s an example of how deployments to this system might work. In this case, multiple nodes are running the Blue version of the application on port 8080 and the Green version of the application on port 8081.

Before the deployment begins, customers are using version 5 of an application on the active Green environment. Version 4 is running on the inactive Blue environment.

Step 1: Version 6 of the application is deployed to the inactive Blue environment and tested.

The blue-green environment just after a new version is deployed to the inactive environment

The blue-green environment, after the new version has been deployed to the inactive environment

Step 2: On a single node (the canary node), the load balancer routes traffic to the Blue port, sending customers to version 6 only on that node. Other customers continue to use version 5 on the other nodes.

The blue-green environment, with one canary node switched to the new version

The blue-green environment, with the load balancer sending traffic to the new version on the canary node

Step 3: If the canary node performs as expected, the load balancer sends traffic on all nodes to the Blue port. Now all customers are using version 6. The Green environment is inactive and ready for a new deployment, to repeat the process.

The blue-green environment after a deployment, with the new version enabled

The blue-green environment after the deployment is complete

Implementing the Blue and Green environments

The way these logical environments are implemented relies on the way that IBM UrbanCode Deploy represents target environments as resources. Instead of duplicating everything, I can install a single production environment, consisting of one or more nodes, and create duplicate resources to represent those logical environments. These resources behave like separate systems on the IBM UrbanCode Deploy server, even though the resources point to the same physical systems.

The following figure shows the resource tree on my server, with a folder for my Blue environment and a folder for my Green environment. I’ve added the same agents to both folders, creating separate agent resources for each environment. This way, the server treats these agents as separate logical environments, even though the Green and Blue agent resources point to the same physical computers.

I’ve also tagged the agent resources with different resource tags. One node is the canary node; this node always gets updated first. The other nodes are updated when I’m satisfied that the canary node is working properly. For more about canary nodes, see Deploying applications with canary nodes.

The resource tree of the blue-green environments

The resource tree, including the resources in the Blue and Green environments

Now I can set up the environments on the server with these separate groups of resources. Here’s the Blue environment:

One environment of the blue-green setup, showing the agent resources that are used in that environment

The resources in the Blue environment

I’ve pulled in the Blue agent resources and mapped my components to them, on all three nodes. In this case, I’m using component tags so I don’t have to map all of my components individually; the component tag grabs all of my components at once. I’ve also added the load balancer component to the nodes so I can call processes to update the load balancer as part of my regular deployment processes. The Green environment is the exact same setup with the Green agent resources.

Because I’m using ports to distinguish my Blue and Green environments, I’ve added an environment property to each environment named LBPort. This property stores the port that the application is running on in each environment; in this case, 8080 for Blue and 8081 for Green.

Finally, I’ve added an application property named activeEnvironment with the name of the active environment. This is just for convenience; I can look it up if I need to know which environment is active rather than examining the load balancer or checking which version my customers are using. It also allows me to prevent processes from running on the wrong environment; I wouldn’t want someone running a process on the environment that my customers are actively using.

Implementing the load balancer with HAProxy

Setting up load balancers for production environments is beyond the scope of this article, but here’s how I set up a simple HAProxy load balancer for the purposes of testing my blue-green deployments with the sample application.

I installed HAProxy on a Linux server and set it up to point to my nodes. Here is the significant part of the haproxy.cfg file:

listen JPetStore 0.0.0.0:80
balance roundrobin
option httpclose
option forwardfor
server servera tpm-bg-a.rtp.raleigh.ibm.com:8081 check
server serverb tpm-bg-b.rtp.raleigh.ibm.com:8081 check
server serverc tpm-bg-c.rtp.raleigh.ibm.com:8081 check

The last three lines point to a certain port on my three nodes. Now I can start HAProxy and go to the URL http://[loadBalancerHostName]/JPetStore; and the load balancer will redirect me to the application nodes. When I want to switch which environment my customers are using, I just update the port numbers in that file and restart HAProxy.

There are many ways to update the load balancer automatically. To make things work natively from within IBM UrbanCode Deploy, I’ve added my load balancer as a component within the JPetStore application. That way, I can call processes on it easily from other processes. That meant first installing an agent on the load balancer host. Then, I created a component to represent the load balancer. This component has no versions; it just has an operational component process called “Switch one node’s port” that changes the active port on a node. To that process, I pass the host name of a node and the port that should be active. In this case, I’m using the agent property agent.Hostname to get the host name of the node that is being updated and the environment property environment.LBPort to get the logical environment’s port.

Component process that updates the load balancer

The component process that calls the generic process that updates the load balancer

Component process step that updates the load balancer

The properties of the component process step that runs the generic process

This component process calls a generic process that runs on the load balancer itself. This process runs a Linux sed command that uses a regular expression to edit the hapyoxy.cfg file based on the input. Again, there are many ways to update the load balancer, but my process accepts the properties Node host name and Port to use, and then uses them to run the following shell script:

sed -i 's/${p:Node host name}:808[[:digit:]]/${p:Node host name}:${p:Port to use}/g' /etc/haproxy/haproxy.cfg

systemctl restart haproxy

I’ve also added a lock step to the process to prevent multiple processes from trying to edit the configuration file at once.

Generic process that updates the load balancer

The generic process that updates the load balancer

Generic process step that updates the load balancer

The generic process step that updates the load balancer

The reason I call a generic process to update the load balancer configuration, rather than doing it in the component process, is that the components are mapped to the nodes, not to the load balancer. Thus, the component processes are running on the application nodes, not on the load balancer. So it’s convenient to call a generic process that can have a new resource context, and in this case it is the resource that hosts the load balancer. Your load balancer might be more sophisticated and have an API that lets you make changes to it remotely.

Sample application

I’m using the JPetStore sample application that you can see in the tutorial Deploying a simple web application on IBM Knowledge Center. That tutorial covers how to set up environments for the application and how to create component processes to deploy the components. The only change I’ve made to the sample application for blue-green deployments is to edit the index.html page of the web component to show the version on the home page. This way, I can tell what version of the application is running without leaving the server.

Modified home page of the JPetStore application

The modified JPetStore home page

Automating the deployment process

Now that I have the application, components, environments, and load balancer set up, I can set up the automation for the blue-green deployment itself. Here’s the application process that I’m using to deploy the new version of the components, update the load balancer, and verify that everything is running properly:

Application process that deploys to an environment and switches the load balancer to it in stages

The complete application process that runs a blue-green deployment

The major steps in this application process are:

  1. A Switch step compares the activeEnvironment property with the name of the environment on which the process is running. If those names match, that means the process is running on the active environment, and to maintain a zero downtime deployment, I don’t want to allow that. If those environment names match, the process ends. There are other ways to ensure that the process is running on the correct environment, but this is how I do it in this example scenario.
  2. An Install Multiple Components step deploys the components as usual. Again, in this case I’m using a component tag to install all of the components in one step. Note that this step runs on all of the nodes in the logical environment.
  3. The process runs automated tests on the newly deployed components. This way, I can verify that they work on the production environment before any customers have access to them.
  4. If either the deployment step or the automated test fails, the process rolls back the newly deployed components to the version that was deployed at the beginning of this application process.
  5. Then the process enters the For Each Tag loop. Remember, by the time we get to this loop, the new version of the application is already deployed to the inactive environment; the only purpose of this loop is to update the load balancer in stages. In the loop, the canary tag runs first, followed by the mainSystem tag. Here are the properties for the For Each Tag loop as I’ve set it up:
    Properties of the For Each Tag loop

    The properties of the For Each Tag step

    1. Within the loop, the first step switches the active port on the load balancer by running the load balancer component process that I covered earlier. This step has no interesting properties because the component process can access the properties it needs directly, without accepting any passed values.Now, for the first time, customers are using the new version of the software. However, on the first time through the loop, the process updates the port for only the canary node or nodes.
    2. The process runs automated tests on the node or nodes that are affected by the tag. If those tests fail, another step reverts the load balancer.
    3. Finally, the process creates a manual approval so I can verify that the canary node is working properly before the loop runs again and updates the other nodes. Your approval process may be different; I’ve used a manual approval so I can manually verify that my deployment is running the way I expect for this example scenario.
  6. At the end of the process, the process updates the active environment property with the name of the newly active environment. I’m using a component process on the load balancer component for this, but there are other ways to change the application property.

Other uses of these strategies

This is only one way to implement blue-green deployments in IBM UrbanCode Deploy; your applications may have different requirements, deployment techniques, environment setups, and layers of testing and approval. However, being able to run automated tasks according to tags and agents in the For Each Tag and For Each Agent gives you flexibility to plan and run complex deployments in a wide variety of situations.

For example, you could run rolling deployments much in the same way. In this case, instead of using a For Each Tag loop to deploy to a canary node first, and then the rest of the environments, you could use the For Each Agent loop to deploy to a few nodes at a time.

I’m very interested to hear how you may have implemented blue-green deployments or how you might want to make them work, so please contact me if you have comments, suggestions, or questions.

Related information