Database Refactoring with Liquibase Michael Le Feuvre, OnBelay Consulting Agenda • What is Database refactoring • Introducing Liquibase • Liquibase example • Questions OnBelay Consulting 2 Database Refactoring Evolutionary change OnBelay Consulting 3 Traditional approach • Traditional approach to changing the database schema. • Make an appointment to see the DBA. • Plead that you want to add a column to an existing table • DBA considers your request • Decides to grant it but first must update the ER diagram in a horribly expensive Database Architecture and Design tool that maintains a logical data model. The model is then rendered in to a physical model. Great for wall charts. • DBA generates or hand-codes the database changes into SQL scripts. • DBA then updates each database directly that requires the changes. OnBelay Consulting 4 Consequences of traditional approach • Because the overhead in the traditional process is so onerous, many organizations required that database design be done up front so that the database was “ready” for programmers to use. • Database artifacts may not be in source control. • If database objects are in source control they aren’t necessarily in the same repository as the application source code. application code changes and database changes are not committed together in the same commit. • The above process is entirely antithetical to an Agile development process and since the database requirements must be known in advance of starting development • Most organizations manage database changes somewhere between traditional and database refactoring. OnBelay Consulting 5 Database refactoring • Based on the concept of refactoring popularized by Martin Fowler: • described as a “refactoring as a small change to your source code that improves its design without changing its semantics.” (Refactoring, Martin, James, 1999) • Evolutionary approaches to database maintenance have been discussed since the 2000’s. • Database refactoring was first described in “Refactoring Databases: Evolutionary Database Design”, Pramod Sadalage , Scott Ambler, 2006. • Database refactoring advocates an evolutionary approach: make small non-breaking changes to the database as you refactor and extend your code. • Developers must be able to manage their own databases. • Requires a change in DBA mindset. The development team initiates the change in consultation with the DBA. • On of the key requirements in database refactoring is that each developer has his or her own database schema. OnBelay Consulting 6 Traditional vs Database Refactoring • A new feature requires a field to be added to the Employee form. • Traditional Approach • DBA creates a script to alter the table to add the new column • DBA runs the above script to update the developers’ databases • Developers make changes to the application to get the development environment to work again. • If everything appears to be working in Dev then the DBA runs the same scripts into Test and the developers promote their code into test as well. • This approach does not support small incremental and concurrent changes. • Database Refactoring Approach • A developer is responsible for both the application code and database script changes to implement the new feature. • The database script and application code changes are checked in together into the same commit. • Test is updated automatically when the feature is committed and the test environment is ready for testing. OnBelay Consulting 7 Refactoring Categories • Not all database changes have the same impact. • Changes fall into the following categories: • Extending the database • Usually a non-breaking change • Example: Adding a table or a nullable column • Altering the database – possibly non-breaking • Possibly a breaking change. Depends on the data • Example: Changing a nullable column to non-nullable • Altering the database – breaking • These are changes that will require changes in the application and/or existing data. • Example: Renaming a table and or column OnBelay Consulting 8 Refactoring approaches • Handling the Altering Databases categories may require: • Scan of existing data to see if there are any data issues. (making a nullable column non-nullable.) • Dividing the required changes into three parts: • First part extends the database by adding a new table, column etc. • Second part alters or copies data • Third part makes the breaking change • Example changing the name and datatype of an existing column • First part: add new column with new name. • Second part: copy data from old column to new column with any required changes. • Third part: delete the old column OnBelay Consulting 9 Challenges to implementing database refactoring • Source code commit should contain both application and database code changes. • An update script should update the database automatically. • The script should be re-runnable • The script should support concurrent independent changes by different developers. OnBelay Consulting 10 Database refactoring tools • Roll your own • Specific tools from the DB vendor – tend to make changes in place. • Flyway • Liquibase OnBelay Consulting 11 Liquibase is a database refactoring tool supported by the Liquibase under the stewardship of Datical. Liquibase Liquibase Pro is the professional, licensed version that extends Liquibase to support additional http://liquibase.org features such as targeted, database rollback, db source objects, etc. OnBelay Consulting 12 Liquibase features • Supports database refactoring either by: • Current to desired state • Diff the desired state (database one) with current state (database two) and generate change log • Apply the change set to current state (database two) to update to desired state • Evolutionary changes • Start with an initial change set that creates the database schema • Create change sets that alter the current database to the next desired state. • Liquibase supports all major databases and may be customized to support any specific db syntax. • In either case Liquibase uses a database logging table to insure that only missing changes are applied to the target. OnBelay Consulting 13 Running Liquibase • Liquibase is packaged as a java jar application with scripts. • Liquibase may be run from: • command line • Maven, • Gradle, or, • from a Java Spring Boot application. • Example: liquibase update OnBelay Consulting 14 Liquibase Commands • Summary of commands: • generateChangeLog – generate a change log from an existing database. (may be used to move from one DBMS to another.) • diff – compare two databases • diffChangeLog – compare two databases and generate a change log to get from the first to the second. • Update – apply the changes • Snapshot – create a snapshot for use in comparing two databases. (its faster to use a snapshot for one of the databases in the comparison.) • Rollback rollback the changes. OnBelay Consulting 15 Database refactoring with Liquibase • Liquibase supports database refactoring through database change logs. • Each log identifies a specific database change. • Liquibase uses a table in the target database to track what changes have been applied to the database. • Liquibase may also be used to create a new database from a snapshot or update an existing database. • Liquibase supports the ability to rollback a change. • Liquibase supports concurrent development – independent changes may be updated in any order. OnBelay Consulting 16 Targeting multiple databases • Most database changes may be specified generically – Liquibase will generate the appropriate SQL • If specific database syntax is required then Liquibase supports multiple ways of specifying the differences: • Properties for each DBMS • Embedded or file based SQL reference by the change set • Specific database types • For example Column types in changes may be specified by: • Using specific SQL: type=“varchar2(255)” • Generically: BOOLEAN, BLOB, DATE, DATETIME, BIGINT – converted to correct DBMS • Using java types: • Java.sql.Types.VARCHAR(20) – Will be converted to VARCHAR2 for ORACLE OnBelay Consulting 17 Liquibase databaseChangeLog • The DatabaseChangeLog defines what changes must be applied. • May be one large file with all changes but is typically broken into a hierarchy of databaseChangeLogs using includes. • The lowest level databaseChangeLog identifies the changes for one feature. • The change is captured in a ChangeSet – typically one changeSet per file. • Supported formats: • XML • YAML • SQL • JSON OnBelay Consulting 18 Typical Liquibase project • Liquibase is usually set up using master and category databaseChangeLogs that include the specific change logs. • MasterChangeLog.xml • Snapshot.xml • createTables.xml • Create_employee.xml • Changes.xml • Feature_xxx.xml • Create_profession_code_table.xml OnBelay Consulting 19 Typical Liquibase workflow • Developer One: • Create a change log to capture the change you want to make. • Create or modify a parent change log to hold all the changes. • Make programming code changes • Test • Commit to feature branch • Developer Two: • Pull latest changes • Run the Liquibase update script to update the database with Developer One’s changes OnBelay Consulting 20 Liquibase example: Add current profession • Your business product owner has asked you to add a current profession field to the employee form. • The product owner has supplied you with a list of professions including the value ‘UNKNOWN’ to be used in case the current profession is hasn’t been collected. • The current database is being managed using Liquibase change logs. OnBelay