Data Management Automation for CI/CD with NetApp and Apprenda

By Atos Apprenda Support

This blog was co-authored by Bikash Roy Choudhury, NetApp

The Problem

Database is often one of the most overlooked and complex parts of the Software Development Life Cycle (SDLC). It is critical for quality testing to have clean and realistic data. Setting up a production-like database for testing purposes is typically a manual and time-consuming process. It involves data copies, migration scripts, obfuscation of sensitive data, and multiple meetings between application developers, DBAs and IT – people who do not speak the same language.

Once a copy of the production database for Dev/Test is created, it quickly becomes “polluted” and less reliable with every test run. The more it is touched by instances of the applications under development, the greater the discrepancy between the reality and the desired state of the data, due to errors in the code base. When developers fix the implementation issues, it is often necessary to restore the test data to its original state right before every new or updated instance of the application is deployed. This requirement increases the amount of complexity and overhead exponentially, especially in the organizations with frequent CI/CD practices. Because of these complexities and the time that it takes to produce fresh data for the preproduction purposes, organizations often forego this process, which puts the code quality at risk and and impacts the delivery.

The Solution

NetApp and Apprenda have created an automated solution that simplifies the process of generating multiple copies of production like database on-demand for pre-production purposes. The cloning of a pristine database for Dev/Test is instrumented automatically at application deployment time, using native ONTAP features like snapshots and FlexClones. The solution is enabled through a direct integration between the Apprenda platform and NetApp SnapCenter REST APIs. It works with a wide variety of DB engines. We use MySQL in the demonstration.

At the compute layer, the integrated solution relies on Apprenda’s deployment policy engine and the abstraction model that frees developers from the need to know intricacies of database cloning and reduces the need in meetings. Apprenda’s policy engine allows the operators to securely segment the platform based on various needs, with SDLC environments being one example. The established policies determine where application instances are deployed and what databases they are connected to. The databases similar to the applications themselves are segmented based on the deployment policies, ensuring secure separation of pre-production from production environments.

Apprenda also takes the concept of “serverless computing” down to the data tier. Multiple servers can be setup to host cloned databases, in which case Apprenda will be controlling the placement of clones based on  the CPU and memory utilization.

At the data management layer, the solution hinges on the capabilities of ONTAP to provide on-demand, instant, thin provisioned clones of the databases that an application can quickly connect to for development and testing purposes. Through policy automation between the Apprenda platform and NetApp SnapCenter REST APIs, data management operations, including application consistent backups, cloning, mounting, starting the database automatically and instant DB restore, are performed with zero storage touch.

The database cloning is governed by Apprenda deployment policies and metadata, which is defined and configured by Apprenda platform operators.

For example, here the platform operators can specify what databases are available for cloning, the location of the production DB, various cloning policies, and the set of servers where the clones can be mounted. The operators also pre-configure several metadata properties that may be set by the developers or DevOps prior to the deployment. DB Cloning type is one of them.

By simply setting this property Developers can influence how the cloning is performed without the need to know any details of the process. There are three options:

  • CloneOriginal uses the original version of the database (a production copy or a specially prepared data set) for the cloning. The application consuming the database gets the most up-to-date data in this case.
  • RestoreClone makes an attempt to restore the last available snapshot of the cloned DB.  This option is set as a default because the restore process is much faster than the initial cloning of the original.

In both of these cases, the application starts from a clean slate as far as the data is concerned. All changes to the data that were made while the application was tested are overwritten by the recent production data or the original state of the earlier clone.

In some cases, however, it is desirable not to refresh the data.

  • KeepExisting option allows developers to maintain the state of the data between the test runs.

This automated database provisioning is conducted in two steps. First, the Apprenda Extension for SnapCenter is invoked by the Apprenda Deployment Pipeline. It communicates with SnapCenter REST API to perform the desired type of cloning and mounting based on the metadata that accompanies the application. At a later stage of the pipeline, Apprenda Bootstrapper for SnapCenter is called to update the application configuration file with the new connection information, so that the application, once it is containerized by the platform, can connect to the cloned database.

Watch a full demonstration of the solution in the video, below. This integrated solution can serve as an enhancement to the existing CI/CD automation. It helps to increase the quality of testing by automatically refreshing the application data without delaying the verification process through manual data migrations. It helps minimize the number of meetings between developers, IT and DBAs, and reduces internal friction from negotiations about DB cloning. It also provides a secure and reliable way to fully bootstrap application instances for pre-production verification purposes.

Atos Apprenda Support