7 ways businesses can improve the quality and speed of database releases
The quality of database releases relies on thorough initial testing in development with teams needing access to fast, reliable data. While database testing is one of the most challenging aspects of development, processes which utilize database clones can overcome these issues.
A database clone is a complete and fully functioning copy of a database that is identical to the original in terms of the metadata and the data. The big difference is that the clones are a fraction of the original size of the database, and so can be created and reset in seconds.
Clones work by exploiting data virtualization technology built into the underlying platform (such as Windows or Kubernetes). A full-sized copy of the original database is captured and, from this data image, virtualized copies or clones can be created and distributed very quickly, even for terabytesized databases. These clones give developers the freedom to quickly create, test, break and reset database environments, without affecting anyone else, or any other process.
This article explains how these ‘ephemeral databases’ can be used to improve development and testing processes in ways that reduce the number of bugs entering the deployment pipeline, drive up the quality of database releases, and so improve the reliability and speed of database deployments to production.
The 7 steps to making the most of clones
1. Enable dedicated development databases
Traditionally, databases have been regarded as a heavyweight asset. Making a copy of a very large database for development or testing work, or refreshing an existing copy, takes many hours, and requires a lot of disk space. Clones, by contrast, have a very light footprint. For example, a clone of a 1TB database can be created or refreshed in 30 seconds rather than many hours, and requires only megabytes of storage space, allowing each developer to work on an isolated ‘sandbox’ database, without fear of disrupting the work of others.
Developers can reset a clone as required, meaning that they can experiment, try out new ideas and run destructive tests. If a test or experiment messes up a database, then spinning up a new clone, or environment, takes seconds. This sort of creative freedom can significantly improve the quality of the resulting code.
2. Prevent problems reaching production
Database tests have always been difficult and very time consuming to set up accurately, meaning testing is often a slow, separate process, performed exclusively by specialist testers, after the developers have a release candidate. By using clones, teams can do more effective database testing, earlier, while they are developing the code. This is often referred to as shift-left testing.
If developers run database tests as well as code quality and coding standards checks, before even committing the code, then it prevents problems ever reaching the main branch of development. Developers can test their migration scripts thoroughly by spinning up a clone from the latest data image, applying the branch migration script, and then running all the necessary tests to ensure it works as expected. All this reduces the risk that merging the branch into the development pipeline will introduce errors or bugs that subsequently break the build.
Catching bugs early in development reduces the cost and time to fix issues and is the most effective way to improve the quality and reliability of database deployments.
3. Run rapid, parallel database testing cycles
Any database test cycle will typically need to: 1) create a copy of the database including the data; 2) run the test and assess the result; and 3) tear down the test and spin up a new database environment ready for the next test. With a large database, steps 1) and 3) take too much time. Also, tests don’t generally pass the first time. There are usually a series of failures, corrections and reruns until the code passes. This can mean creating and breaking a database many times.
With clones, this setup-test-reset process is rapid and lightweight, meaning that much more extensive and effective testing can be achieved in the same testing window. Developers and testers can now perform rapid series of test cycles and spin up multiple copies of a database, to run tests in parallel.
4. Improve testing accuracy with realistic test data
Certain types of tests, like performance tests or UAT tests, require realistic data. By using clones, data images can be created that accurately reflect the production system, in data characteristics, volume and distribution, while supporting compliance, either by a process of masking the production database, or by realistic data generation.
Clones with sensitive data obfuscated can then be deployed to all development and test systems that require them, quickly and efficiently. Use of realistic data during development, and in integration tests, will help development teams find data compatibility issues earlier in the release pipeline, reducing the occurrence of data-related bugs.
5. Deploy clones in CI pipelines for continuous verification and testing
Clones are designed to fit into CI processes, where the resource, in this case a database, can be spun up for a quick, specific task, modified as required, and then destroyed and recreated. This means that teams can build a CI pipeline that automates database builds throughout the development process, continuously verifying that the source code and data files can be used to successfully create the database, with its development data, to the correct version.
These practices will drive up the quality of the changes delivered and reduce the likelihood of a failed build.
6. Reproduce difficult production bugs, and develop fixes, faster
Improving the quality of database releases also means diagnosing any production bugs quickly and efficiently. Some problems are very difficult to replicate outside the production environment, and without the production data. With clones, the data images can be masked automatically, at the point of creation, so teams can develop and test fixes in a secure environment, on clones with real, but obfuscated, data.
7. Enable collaborative branch-based development
With clones, it becomes viable for teams to split each item in the development backlog (new feature, update, bugfix) into a task-based branch, supported by a dedicated database (a clone).
The ability to separate tasks in this way means a developer, or small team of developers, can focus on each specific end-user requirement, leading to higher-quality code. It also encourages DevOps collaboration: with potentially disruptive database changes isolated from other ongoing work, Dev and Ops can work on them in partnership. This left-shifts operational knowledge into the development cycle to fix problems, optimize performance or tighten security. All of which leads to higher quality code and fewer deployment problems.
Conclusions
Ephemeral, or disposable, databases enable agile development and testing practices that entail frequent rapid cycles of database development and testing. Developers catch bugs earlier when they are much easier and cheaper to fix. They can work on isolated databases, one clone per branch of development, if required, meaning experimental and destructive changes and tests can be run without fear of disrupting the whole development effort. Changes that affect the data model or data, or that have security or performance implications, can be isolated in a branch, and resolved collaboratively with operational expertise. The resulting code will be higher quality and much safer to deploy without risk of causing disruption to live services.
By incorporating clones into CI pipelines, teams can continuously verify a working database system throughout development. By testing with realistic data, they can find data-related bugs quickly and have confidence that the behavior seen during testing will reflect accurately what will be expected in the production system.
Want to put cloning into action? Find out more about our cross-database tool, Redgate Clone.
Read next
Blog post
Redgate opens the doors to cross-database DevOps Test Data Management
A constant challenge for many development teams is how to include test data management in the development process. Testing with consistent, realistic and compliant datasets, and bringing the database into CI and DevOps pipelines, has been shown to catch data-related issues long before they reach customers. The result: an increase in efficiency, a reduction in
Blog post
The ten habits for highly successful end-to-end Database DevOps
Database DevOps has come of age. Now seen as a key technical practice which can contribute to the successful implementation of DevOps, it stops the database being a bottleneck and makes releases faster and easier. Conversely, perhaps, the automation and audit trails it introduces can help to protect personal data within databases and make compliance
Tools in this post
You may also like
Blog
Using the new Flyway Diff commands
Blog
What is DORA, and how does it impact the database?
Blog