cloud lifegurard
CPM Cloud Protection Manager
Enterprise-class EC2 Backup
and Disaster Recovery
 
TRY CPM FREE FOR 30 DAYS
Type your first name

Type your last name

Please let us know your email address

Please type your company name

Please type your phone number

By submitting this form you agree to the Terms of use

How to Integrate Automated Cloud Backup in your DevOps Pipelines (CI, CD) - N2WS blog

Written by  N2W Software Team Thursday, 11 May 2017

blog-integrate-automatic-backup-devops-pipeline Think of an application which goes through development, followed by staging and finally the production process. With different kinds of sanity checks, source code management as well monitoring and rollback needs, if your development workflow doesn’t have a CI/CD system, chances are that your team has to deal with builds which are inconsistent and unpredictable, with random issues that are generally hard to track down and fix. For instance, there could be a feature that works on the developer’s machine, but doesn’t work as expected in a different environment.


What is Continuous Integration (CI)?

Continuous Integration is the process of running a series of steps on your project to make sure that it compiles without error, passes all test cases, and generates a working package without any issues. Each change in software generally goes through a CI pipeline.

What is Continuous Delivery (CD)?

This process takes CI a step further. Once the build is generated, it has to be deployed on servers, or published on a mobile store. Continuous Delivery pipeline allows you to define steps to deploy build artifacts on servers such as staging or production. Once the build is generated and has passed all test cases, it is deployed on development or testing environment where it can be tested further to ensure that the build is as expected.

Why it is Important?

When building software, it is very important to take care with deployment of each incremental change so that delivery is flawless and predictable. Automation plays an important role in ensuring that each software release goes through a series of clearly defined steps. CI systems like Jenkins, Travis-CI, and CircleCI enable you to optimize this workflow. Jenkins is the most widely used tool for setting up CI/CD pipelines.

As each incremental change goes through CI/CD pipeline, problems may be identified early in the development lifecycle. You can set your development workflow in a way that allows you to be alerted to any integration, packaging, or sanity-related issue after the code is pushed into source code repository.

How Does it Work?

The CI /CD process can be customized based on the type of project you’re setting it up for. Let’s understand each step:

  • Sanity check (linting): This should be the first step. It will ensure that changes in source code conform to predefined coding rules and that there are no syntactical errors.
  • Compile: The compilation step sees to it that all libraries and dependencies are in place and that there are no issues in generating binaries.
  • Packaging: This step will create a bundle with all binaries along with static resources and dependencies so that we can deploy the application to the server. In case of mobile apps, it can also sign packages with developer keys to ensure integrity.
  • Unit test: This step will execute all unit tests on the build. Application will be ready for deployment pipeline once all tests are passed.
  • Backup: This is a very crucial step that involves backing up existing database, configurations, and data files before deploying any build.
  • Deployment: Once a backup is completed successfully, the application will be deployed, replacing the existing one on the server. This step can also take care of any process you might want to set up to ensure zero downtime deployment.
  • Smoke test: After the application is deployed, it’s ideal to have a series of automated sets of basic functionality tests to ensure that system is up and functional.

 

Additional things to consider during implementation:

  • Protecting data and configuration related files is very important. Having an effective backup strategy is the key to implementing the CD pipeline which keeps maintaining a reliable set of backups.
  • Here are a few things to consider when implementing the backup stage:
    • As our objective is to iterate faster, there can be multiple builds daily. This results in huge storage requirements for backups. However, as changes between these builds won’t be major, we can save significant amount of storage by backing up only delta between builds.
    • Having backup is great. However a backup you cannot restore is pointless. It’s essential to have periodic verification of backups, which again can be made a part of a backup pipeline.

 

How to Implement a Robust CI/CD Pipeline

Although the exact implementation of CI/CD pipeline depends on the kind of project and the complexity of build/deployment process, we’ll cover some basic aspects of setting up a robust CI/CD pipeline.

how-to-implement-CICD-pipeline

Sanity tools: Linting and static analysis tools are quite useful at this stage. Every language comes with a set of linting tools that allow you to define code formatting rules. This process will highlight any potential issues that might be in the source code. Some commonly used tools are cppcheck for C/C++, checkstyle for Java, and JSLint for JavaScript. SonarQube is another tool which does extensive analysis of source to identify quality issues with source code.

Build & package tools: Use project/programming language specific build and package tools like Gradle for Android project, Maven for Java based project, npm, and pip for NodeJS/Python projects.

Unit testing tools: JUnit, Selenium, and Mockito can all be used to run tests and generate reports. Jenkins provides a lot of built-in unit testing plugins to run tests. Almost any shell script that offers you flexibility to run any unit testing program can be executed.

Automated backup - at what stages of integration and delivery?

It’s a good idea to run an automated backup to capture a snapshot of existing deployment before delivering a new build. Solutions such as Cloud Protection Manager are built from the ground up for AWS cloud backup and Amazon EC2 and provide flexible and scalable snapshot management, automation and monitoring. Backup should be run right before deploying to production environment. The best way to implement this step is to use a tool like Duplicity which allows you to efficiently store backups by calculating delta between changes and storing it in a secure way. Check out this link for more information on how to set it up.

Import artifacts to backup include:

  • Compiled binary
  • Test reports
  • Build logs
  • Configurations
  • Data files

 

Deployment tools: Jenkins offers plugins to deploy code via SSH, FTP, etc. AWS also offers a service called AWS CodeDeploy which allows you to deploy builds on Amazon EC2 instances. AWS offers AWS CodePipeline this is one integrated tool for CI/CD need.

Benefits of Automated Build Backups

Easier rollback than previous version

The biggest advantage of having automated backups in place is that they allow you to set up a CD pipeline in such a way that it becomes very easy to rollback to previous versions in case things go wrong. Having this capability is priceless; you can set up a backup pipeline as part of the CD process. This will back up existing configurations, data files, etc. on a service like the S3 or a similar file storage service.

Refer older build logs

When things go wrong, you can always refer to build logs from earlier builds and compare them with the current build to find out any issues in the current build.

Compare performance with older builds

Having the ability to track all build data helps you compare important metrics to make sure there is no dramatic impact on binary size, performance, or deployment time due to a change in code.

Keeping track and referring configurations used

Configuration changes can have a huge impact on functionality and having backups for configurations can be a lifesaver. Rolling back to an earlier revision becomes much easier.

Conclusion

In a nutshell, a well structured CI/CD implementation is an essential part of the software delivery process. It enables teams to move faster and focus on delivering features without worrying about breaking something. It doesn’t matter if your team is working on a mobile app, firmware, or cloud project, the CI/CD system will greatly improve workflow and quality of deliverables. With this information, you can go ahead and implement CI/CD system for your organization.

Start Protecting Your Cloud Deployment Properly

Start your free trial today to ensure implementing an automated robust, scalable, enterprise-class cloud backup and recovery solution