Extracting and importing data for development and testing is made trickier due to issues such as constraints, dependencies, and special data types. This article introduces a cross-RDBMS solution with JSON for data storage and PowerShell cmdlets that use ODBC to help automate extraction and import, and JSON Schema for validation. Just provide a DSN, and you’re good to go! Read more
The management of datasets is intrinsic to effective team-based database development. This article reviews what's required of test datasets in database development and then proposes a system of managing them in a migration-based approach that adopts a common standard for their storage, using JSON. Read more
With Flyway, you can adopt a test-driven development strategy that will allow you to test and evaluate databases, and database objects, at every phase of the database development lifecycle. The further down the delivery pipeline that bugs appear, the more costly in time and resources they are to fix. This approach will allow you to catch many of them before the database change even gets committed to version control, making a continuous delivery process much easier to adopt and sustain. Read more
This article will help you understand the steps to better test data management, when using a migrations-based approach to database development and deployment. It explains the different types of data required and why, the need for separation of DDL and DML code, and the most efficient way to create, load and switch between the different required data sets. Read more
An overview of the challenges of database testing and test data management, reviewing the different types of database test that need to run during development work, what sort of test data they require, and how to manage all the required data sets, during development, in a way that allows rapid cycles of parallel testing. Read more
A cross-RDBMS way of exporting, deleting and inserting data, for database development work. It is a PowerShell automation technique for Flyway that uses JSON files for data storage. It should help a team maintain datasets between database versions, as well as to switch between the datasets required to support different types of testing. Read more
Before you get very far with database development and testing, you need to be clear about your strategy for handling data. In this article I'll explain some of these issues in general terms, and then demonstrate how you can navigate these problems easily with Flyway. Read more
The test data management strategy for any RDBMS needs to include a fast, automated way of allowing developers to "build-and-fill" multiple copies of any version of the database, for ad-hoc or automated testing. The technique presented in this article uses a baseline migration script to create the empty database at the required version, which then triggers a PowerShell callback script that bulk loads in the right version of the data. Read more
How to generate "realistic but fake" SQL Server test data, using SQL Data Generator, and then use SQL Data Compare to produce an INSERT script that Flyway can run to load the data into the database. The technique is useful for managing small volumes of test data or for "topping up" existing data when you create new entities. Read more
This article provides SQL "bulk copy" routines to extract data from and load it into a SQL Server database, rapidly, and demonstrates a way to automatically bulk insert the test data in a Flyway build, using a SQL callback. Read more