Database Refactoring with Liquibase

Michael Le Feuvre, OnBelay Consulting Agenda

• What is refactoring • Introducing Liquibase • Liquibase example • Questions

OnBelay Consulting 2 Database Refactoring

Evolutionary change

OnBelay Consulting 3 Traditional approach

• Traditional approach to changing the . • Make an appointment to see the DBA. • Plead that you want to add a column to an existing table • DBA considers your request • Decides to grant it but first must update the ER diagram in a horribly expensive Database Architecture and Design tool that maintains a logical data model. The model is then rendered in to a physical model. Great for wall charts. • DBA generates or hand-codes the database changes into SQL scripts. • DBA then updates each database directly that requires the changes.

OnBelay Consulting 4 Consequences of traditional approach

• Because the overhead in the traditional process is so onerous, many organizations required that be done up front so that the database was “ready” for programmers to use. • Database artifacts may not be in source control. • If database objects are in source control they aren’t necessarily in the same repository as the application source code. application code changes and database changes are not committed together in the same commit. • The above process is entirely antithetical to an Agile development process and since the database requirements must be known in advance of starting development • Most organizations manage database changes somewhere between traditional and database refactoring.

OnBelay Consulting 5 Database refactoring

• Based on the concept of refactoring popularized by Martin Fowler: • described as a “refactoring as a small change to your source code that improves its design without changing its semantics.” (Refactoring, Martin, James, 1999)

• Evolutionary approaches to database maintenance have been discussed since the 2000’s. • Database refactoring was first described in “Refactoring : Evolutionary Database Design”, Pramod Sadalage , Scott Ambler, 2006. • Database refactoring advocates an evolutionary approach: make small non-breaking changes to the database as you refactor and extend your code. • Developers must be able to manage their own databases. • Requires a change in DBA mindset. The development team initiates the change in consultation with the DBA. • On of the key requirements in database refactoring is that each developer has his or her own database schema.

OnBelay Consulting 6 Traditional vs Database Refactoring

• A new feature requires a field to be added to the Employee form. • Traditional Approach • DBA creates a script to alter the table to add the new column • DBA runs the above script to update the developers’ databases • Developers make changes to the application to get the development environment to work again. • If everything appears to be working in Dev then the DBA runs the same scripts into Test and the developers promote their code into test as well. • This approach does not support small incremental and concurrent changes. • Database Refactoring Approach • A developer is responsible for both the application code and database script changes to implement the new feature. • The database script and application code changes are checked in together into the same commit. • Test is updated automatically when the feature is committed and the test environment is ready for testing.

OnBelay Consulting 7 Refactoring Categories

• Not all database changes have the same impact. • Changes fall into the following categories: • Extending the database • Usually a non-breaking change • Example: Adding a table or a nullable column • Altering the database – possibly non-breaking • Possibly a breaking change. Depends on the data • Example: Changing a nullable column to non-nullable • Altering the database – breaking • These are changes that will require changes in the application and/or existing data. • Example: Renaming a table and or column

OnBelay Consulting 8 Refactoring approaches

• Handling the Altering Databases categories may require: • Scan of existing data to see if there are any data issues. (making a nullable column non-nullable.) • Dividing the required changes into three parts: • First part extends the database by adding a new table, column etc. • Second part alters or copies data • Third part makes the breaking change

• Example changing the name and datatype of an existing column • First part: add new column with new name. • Second part: copy data from old column to new column with any required changes. • Third part: delete the old column

OnBelay Consulting 9 Challenges to implementing database refactoring • Source code commit should contain both application and database code changes. • An update script should update the database automatically. • The script should be re-runnable • The script should support concurrent independent changes by different developers.

OnBelay Consulting 10 Database refactoring tools

• Roll your own • Specific tools from the DB vendor – tend to make changes in place. • • Liquibase

OnBelay Consulting 11 Liquibase is a database refactoring tool supported by the Liquibase under the stewardship of Datical. Liquibase Liquibase Pro is the professional, licensed version that extends Liquibase to support additional http://liquibase.org features such as targeted, database rollback, db source objects, etc.

OnBelay Consulting 12 Liquibase features

• Supports database refactoring either by: • Current to desired state • Diff the desired state (database one) with current state (database two) and generate change log • Apply the change set to current state (database two) to update to desired state • Evolutionary changes • Start with an initial change set that creates the database schema • Create change sets that alter the current database to the next desired state. • Liquibase supports all major databases and may be customized to support any specific db syntax. • In either case Liquibase uses a database logging table to insure that only missing changes are applied to the target.

OnBelay Consulting 13 Running Liquibase

• Liquibase is packaged as a java jar application with scripts. • Liquibase may be run from: • command line • Maven, • Gradle, or, • from a Java Spring Boot application.

• Example: liquibase update

OnBelay Consulting 14 Liquibase Commands

• Summary of commands: • generateChangeLog – generate a change log from an existing database. (may be used to move from one DBMS to another.) • diff – compare two databases • diffChangeLog – compare two databases and generate a change log to get from the first to the second. • Update – apply the changes • Snapshot – create a snapshot for use in comparing two databases. (its faster to use a snapshot for one of the databases in the comparison.) • Rollback rollback the changes.

OnBelay Consulting 15 Database refactoring with Liquibase

• Liquibase supports database refactoring through database change logs. • Each log identifies a specific database change. • Liquibase uses a table in the target database to track what changes have been applied to the database. • Liquibase may also be used to create a new database from a snapshot or update an existing database. • Liquibase supports the ability to rollback a change. • Liquibase supports concurrent development – independent changes may be updated in any order.

OnBelay Consulting 16 Targeting multiple databases

• Most database changes may be specified generically – Liquibase will generate the appropriate SQL • If specific database syntax is required then Liquibase supports multiple ways of specifying the differences: • Properties for each DBMS • Embedded or file based SQL reference by the change set • Specific database types • For example Column types in changes may be specified by: • Using specific SQL: type=“varchar2(255)” • Generically: BOOLEAN, BLOB, DATE, DATETIME, BIGINT – converted to correct DBMS • Using java types: • Java..Types.VARCHAR(20) – Will be converted to VARCHAR2 for ORACLE

OnBelay Consulting 17 Liquibase databaseChangeLog

• The DatabaseChangeLog defines what changes must be applied. • May be one large file with all changes but is typically broken into a hierarchy of databaseChangeLogs using includes. • The lowest level databaseChangeLog identifies the changes for one feature. • The change is captured in a ChangeSet – typically one changeSet per file.

• Supported formats: • XML • YAML • SQL • JSON

OnBelay Consulting 18 Typical Liquibase project

• Liquibase is usually set up using master and category databaseChangeLogs that include the specific change logs. • MasterChangeLog.xml • Snapshot.xml • createTables.xml • Create_employee.xml • Changes.xml • Feature_xxx.xml • Create_profession_code_table.xml

OnBelay Consulting 19 Typical Liquibase workflow

• Developer One: • Create a change log to capture the change you want to make. • Create or modify a parent change log to hold all the changes. • Make programming code changes • Test • Commit to feature branch • Developer Two: • Pull latest changes • Run the Liquibase update script to update the database with Developer One’s changes

OnBelay Consulting 20 Liquibase example: Add current profession

• Your business product owner has asked you to add a current profession field to the employee form. • The product owner has supplied you with a list of professions including the value ‘UNKNOWN’ to be used in case the current profession is hasn’t been collected. • The current database is being managed using Liquibase change logs.

OnBelay Consulting 21 Add current profession tasks

• Steps to add current profession code to PR_EMPLOYEE • Create PROFESSION_CODE table • INSERT data into PROFESSION_CODE table • Add CURRENT_PROFESSION_CODE to PR_EMPLOYEE • Update CURRENT_PROFESSION_CODE for existing employees to UNKNOWN • Update the CURRENT_PROFESSION_CODE column to NOT NULL

OnBelay Consulting 22 Create PROFESSION_CODE

OnBelay Consulting 23 Insert profession codes

…

OnBelay Consulting 24 Add current profession to employee

OnBelay Consulting 25 Update existing employees

OnBelay Consulting 26 Make CURRENT_PROFESSION not nullable

/>

OnBelay Consulting 27 Apply the changes

• Assuming a directory structure: Changes.log currentProfession current_profession_changes.xml create_profession_code_table.xml insert_profession_codes.xml … Command: liquibase update

OnBelay Consulting 28 Databasechangelog table id author filename dateexecutedorderexecutedmd5sum description liquibase labels deployment_id 1 lefeuvrem/com/onbelay/sqlcourse/table/create_employee_table.xml 21:41.8 1 8:77bd20ef441e754974cf7a58079f954dcreateSequence sequenceName=EMPLOYEE_SEQ; createTable3.8.9 NULL tableName=PR_EMPLOYEE8.38E+09 1 lefeuvrem/com/onbelay/sqlcourse/table/employee_table_constraints.xml 21:41.8 2 8:c1f4f5e754940b06ed2e130ba4f7a9d6addUniqueConstraint constraintName=ENP_NAME_UC,3.8.9 tableName=PR_EMPLOYEE;NULL 8.38E+09 createIndex indexName=EMP_NAME_IDX, tableName=PR_EMPLOYEE; addForeignKeyConstraint baseTableName=PR_EMPLOYEE, constraintName=EMP_MANAGER_ID_FK, referencedTableName=PR_EMPLOYEE 1 lefeuvrem/com/onbelay/sqlcourse/changes/create_profession_code.xml 21:41.9 3 8:43b057dfb27050b7c9580726a9b68bdecreateTable tableName=PROFESSION_CODE 3.8.9 NULL 8.38E+09 1 lefeuvrem/com/onbelay/sqlcourse/changes/insert_profession_codes.xml 22:54.6 4 8:e371a2b74a8aaaace42a4c265334f0c7insert tableName=PROFESSION_CODE; insert tableName=PROFESSION_CODE;3.8.9 NULL 8.38E+09 insert tableName=PROFESSION_CODE; insert tableName=PROFESSION_CODE; insert tableName=PROFESSION_CODE 1 lefeuvrem/com/onbelay/sqlcourse/changes/add_current_profession_to_employee.xml22:54.6 5 8:238a70a40dd35a5304f4afe437ab0ab0addColumn tableName=PR_EMPLOYEE 3.8.9 NULL 8.38E+09 1 lefeuvrem/com/onbelay/sqlcourse/changes/update_employee_current_profession.xml22:54.6 6 8:25e4caefbc8ce39b28f0bd5124082eb0update tableName=PR_EMPLOYEE 3.8.9 NULL 8.38E+09 1 lefeuvrem/com/onbelay/sqlcourse/changes/make_employee_current_profession_notnull.xml22:54.6 7 8:ed55fd7afb4d49cb82d9ca6f4124e401addNotNullConstraint columnName=CURRENT_PROFESSION_CODE,3.8.9 NULL tableName=PR_EMPLOYEE8.38E+09

OnBelay Consulting 29 Example review

• Each change set described earlier is uniquely identified by: • LogicalFileLocation (preferred over using physical file location) • Id • Author • All the changes could be captured in one databaseChangeLog if desired.

OnBelay Consulting 30 Other change examples

• A change set identifies one or more changes: • createTable • createIndex • createView • dropTable • dropColumn • addColumn • …

OnBelay Consulting 31 Liquibase best practices

• One change set per file, usually (or one per feature) • Use logicalFilePath to define the unique key for the change. (The alternative is that it uses the actual physical file path and that may cause problems if you move files around. • Set objectQuotingStrategy to QUOTE_ONLY_RESERVED_WORDS • Use updateSQL to generate the SQL that will be run to update the database. The SQL generated is easier to review with a DBA.

OnBelay Consulting 32 Liquibase resources

• http://liquibase.org

• If you are interested in Liquibase Pro: • Kevin Colegrove [email protected]

OnBelay Consulting 33 Future OnBelay initiatives

• SQL for Software Developers course • Dagnabit a Java open source library for exploring graphs using a fluent, functional programming style.

OnBelay Consulting 34 Any Questions ?

This Photo by Unknown Author is licensed under CC BY-SA

OnBelay Consulting 35 Thank You

Michael Le Feuvre 250-575-1961

[email protected]

Onbelayconsulting.ca