Automated deployments

Application deployment should not be a manual, error-prone process. Instead, it should be a well understood, automated process which integrates all the steps necessary for deployment. Ideally it can be triggered by the single push of a button, preferably on a build server. Formalizing your deployments like this has several advantages:

  • Everybody can deploy

    Application deployment should be a non-issue. Everyone in the team should be able to do an application deployment since the whole process is formalized in a well defined automated procedure. Deployments should not be limited to a select few persons who happen to have the right knowledge and/or permissions. Instead, application deployment should be the result of an agreement between the development- and operations team. Whoever pushes the deployment button is unimportant as long as there is confidence in the deployment process, responsabilities when things go wrong are well understood and everybody understands and agrees about the deployment steps.

  • Deployments can be done more frequently

    Application deployments are a notoriously stressful time. Will all the parts work together as expected? It is easy to make a mistake if deployment is a manual process where several complex steps have to be executed in a particular order. Because of this, deployments are not done very often and are often painful when they are performed. The solution ofcourse, is to automate this process and start performing it more frequently. If you strive to achieve continuous deployment you will have to automate the deployment process because it will be triggered on every commit.

  • Repetition builds confidence

    Confidence in the deployment process will increase when you deploy more often. At first there will be numerous deployments to the test environment, less to the staging environment and even less to the production environment. As you work towards the goal of continuous deployment the number of deploys to the staging and production environments will go up until you reach the the point where deployments are done to all three environments for every commit (depending on the build and test results). But even if you don't yet practice continuous deployment you will immediately reap the benefits of multiple deploys to the test environment because the deployment procedure will be the same for every environment. This is a key advantage!

In the next section we will look at some tools to help you automate your application deployments.

Deployment tools

Deployment tools range from simple bash scripts to complex (commercial) orchestration tools. Depending on your application you have to evaluate which solution best fits your needs. Generally speaking, bash scripts are probably a little too low-level to allow for maintainable and easy-to-read deployment scripts. On the other hand big orchestration tools might be overkill for most application deployments, especially if those applications are built with scripting languages like php and ruby, which don't require any compilation and / or linking steps.

I mostly work with PHP web applications. This means that deployment can be a relatively easy process compared to full-blown java applications for example (although Fabric can also handle those situations). After reviewing the options I decided to use Fabric for my deployments. Python is an easy and ubiquitous language, available out of the box on most platforms. Fabric provides a high-level abstration library over SSH which suits my needs perfectly. At the moment I only have to connect to a single server to deploy an application. I will probably take another look at the more advanced orchestration tools when our applications become more involved infrastructurally (e.g. multiple application, database and loadbalancing servers).

To avoid creating a seperate deployment script for every application I decided to create a boilerplate Fabric deployment script which only needs the right information filled in to be fully functional. This makes the process of automating our deployments much faster. The next section describes the way I structured the boilerplate and how you can use it in your own applications.

Fabric Deployment Boilerplate Structure

I have created a little boilerplate project to automate application deployments with Fabric. It's called FaDeBo (as in Fabric Deployment Boilerplate :-). It is supposed to be a drop in deployment script that will get you up-and-running with automated deployments in no-time. It will only require a minimal set of configuration. I will explain the project in the following sections. I'd like to remind you that this is not a tutorial on how the Fabric library works. The documentation provided by the Fabric project is really excellent and I encourage you to take some time and read it.

Often you will want to deploy your application to a test, staging and production environment. These are the default environments that are defined in the Fabric Deployment Boilerplate but you can define additional environments if you need them. For every environment you will need to specify a settings.py file with the following contents:

context = {

  # Which user to use for making the remote connection
  'user'            : 'deploy',

  # Specify the GIT repo to use for deployment
  'repo'            : '',

  # The application name
  'app_name'        : '',

  # The application directory on the remote server
  'app_dir'         : '',

  # The web root directory
  'public_dir'      : '',

  # The environments directory
  'env_dir'         : 'deploy',

  # How many previous releases should we keep
  'keep_releases'    : 5,

  # Which server do we want to connect to
  'host_string'     : '',

  # Specify the url for a package on a remote location
  'package_url'     : '',

  # What format is the package in. Can be tar or zip.
  'package_format'  : 'zip', 

  # This is dictionary of symlinks. It tells which files from
  # the shared directory should be symlinked into the new release
  # and in which location they should be put
  'symlinks'        : {},

  # A list of local artifacts that are deployed to the remote location
  # The keys specify the local file and the values specify the remote path
  'artifacts'       : {}
}

The Fabric Deployment Boilerplate expects to be included in your project with the following directory structure:

fabfile.py
deploy/test/settings.py
deploy/test/shared/
deploy/staging/settings.py
deploy/staging/shared/
deploy/production/settings.py
deploy/production/shared/

The fabfile.py is the main deployment script which contains a number of functions that need to be defined based on your application needs. The next section explains the main logic and what you need to customize.

Fabric Deployment Boilerplate Methodology

Bootstrapping

Fabric Deployment Boilerplate bootstraps the remote environment by creating a default directory structure on the remote server. It will create a 'releases', 'src', 'tools' and 'shared' directory. The 'releases' directory contains a number of releases with a 'current' symlink pointing to the most recent one. This enables really quick rollback by just switching the symlink back to a previous release. The 'src' directory contains external source code that can not be downloaded by a package management system. The 'tools' directory contains any global tools necessary to run the application and finally the 'shared' directory contains all the files that are shared between different releases. Any environment specific configuration files are to be placed in this directory.

Fabric Deployment Boilerplate exposes two functions that allow management of the 'shared' folder: getfolder and putfolder. They simple copy (and overwrite) files from your local shared folder (see the above paragraph about Fabric Boilerplate Structure) to the remote location and vice versa.

Deployment

The main deployment code is defined as follows:

# Deploy application to the target environment
@task
def deploy(branch='master',auto_mode=False):
  ...
  # Execute all the deployment steps in order
  with failwrap():
    execute(check_prerequisites)
    execute(deploy_package)
    execute(resolve_dependencies)
    execute(migrate_up)
    execute(pre_activate)
    execute(activate)
    execute(post_activate)
    execute(smoke_test)
  ...

As you can see it runs a sequence of functions to deploy the applications. I think most of them are pretty self explanatory but let's review them one at a time.

  • check_prerequisites:

    If your application dependes on any prerequisites this is the place to check for them. Normally it only checks if the environment is properly bootstrapped.

  • deploy_package:

    This function actually sends your application to the remote server. There are three ways to define your application source. If the repo location is specified in the environment settings.py it will use git commands to create a tar.gz archive of the specified branch / commit and send it to the remote server. If a remote package location is specified it will be downloaded on the remote server for deployment. Finally you can specify local artifacts that are to be copied to the remote location.

  • resolve_dependencies:

    If the applciation needs any dependencies you can download them here, preferably with a package manager like Composer or Bundler.

  • migrate_up:

    Most applications somehow depend on a database. In this function you can define the procedure required to migrate the database structure to the new release. Some frameworks have dedicated tools for this purpose, but sometimes you have to rely on an external library or write one yourself.

  • pre_activate:

    All the steps required before activation of the release are performed in this function. By default it will create the symlinks that are defined in the settings.py file, set all the correct file permissions and stop any running application services. Additional steps may be defined here.

  • activate:

    This simply switches the 'current' symlink from the previous release directory to the newly deployed release directory.

  • post_activate:

    By default this function will start any stopped services and clean up old release directories.

  • smoke_test:

    You want to make sure that your application succeeded by performing a simple to test to check if the application is up-and-running. This can be as simple as a curl on port 80 or elaborate as you want.

Rollback

A deployment can fail at any step in the deployment procedure. We want to make sure that we don't leave the application in an unkown state and we also don't want to leave any failed deployment files on the remote server. The rollback procedure is triggered when a deployment step fails or when it is called manually from the command line. By default it will check which deployment steps have been performed and try to roll them back. Currently these steps consist of switching the 'current' symlink back to the previous deploy and running the down migrations on the database.

Conclusion

The Fabric Deployment Boilerplate has already been incorporated in several of my projects at work. It was mostly a matter of adding the right settings and filling in the necessary steps to allow for an automated deployment. Sometimes you have to add quite a lot of code and sometimes you can get by with just the defaults provided by the Fabric Deployment Boilerplate.