Database Migrations with Liquibase in Spring Boot

This post links to: Building a Spring Boot CRUD App With Postgres from Scratch: The Complete Guide.

This guide follows on from Securing Your API with Spring Security.

Up until this point, everything we've created is mostly fine for production systems. However, when it comes to the database management, we're currently allowing Hibernate to control our Data Definition Language (DDL) SQL statements like: CREATE, ALTER & DROP. This will need to change if we plan on running our application in a production environment.

While using Hibernate for DDL is convenient during local development, it becomes risky and unreliable in production systems where multiple developers, environments, deployments, and databases all need to stay in sync.

In this blog post we will discuss adding a database migration tool for a production ready API using Liquibase with Spring Boot to avoid using Hibernate for our DDL SQL statements.

What Is a Database Migration?

A database migration is the process of moving, restructuring, or upgrading data and its underlying structure. In our case, we care more about the restructuring and upgrading of data and our database schemas. This would include database operations like: creating tables, adding or removing columns, renaming fields, creating indexes etc.

For example, let's say our database in production has a users table like below:

Then let's say our requirements change, and now we want to be more lenient with the length of usernames stored in the database, allowing usernames of maximum 30 characters like so:

We would need some way to migrate the existing data and allow new data to have this varchar(30) field type.

This migration of data could be done manually. However, this could prove complex and also prone to human error. There's also no historical log of changes showing what fields use to be in the past. That's why production systems use database migration tools like Liquibase to store migration scripts as code and apply them automatically on application startup. A migration tool will also keep track of changes made and make sure not to re-run operations if they've already been applied.

Having a migration tool provides:

• Repeatable deployments
• Version-controlled database changes
• Consistent environments
• And overall safer production releases
  • Data is one of the most important parts of any application, we want to try to avoid any data corruption or missing data

Understanding ddl-auto

Currently, in our application, we have defined the following in our application.yml properties file:

spring:
  #...other properties
  jpa:
    hibernate:
      ddl-auto: update

This tells Hibernate to automatically update the database schema based on our defined entity class, like our User.java entity:

@Builder(toBuilder = true)
@Getter
@Setter
@AllArgsConstructor
@NoArgsConstructor
@Entity
@Table(name = "users")
public class User {

    @Id
    @GeneratedValue
    @Column(name = "id")
    private Long id;
    
    //...other fields
}

Hibernate will compare database definitions to the Java defined entity and then generate the required DDL SQL statements to reflect the entity updates in the database.

Some other common ddl-auto property values include:

• none - No schema changes
• validate - Validates schema matches entities
• update - What we are using, attempts to update schema automatically
• create - Drops and recreates schema on startup
• create-drop - Recreates schema and drops on shutdown

While this is useful for development, update has several limitations:

• Changes are not versioned
• SQL is generated implicitly, meaning we lose control
• Renames can behave unpredictably
• Rollbacks are difficult

Overall, production deployments become more risky and unpredictable.

What is Liquibase

Liquibase is an open-source database migration tool used to manage and version-control database schema changes.

So instead of manually updating databases or relying on Hibernate to generate schema changes automatically, Liquibase allows us to define database changes explicitly using migration files called changelogs.

These changelogs describe changes such as:

• Creating tables
• Adding or removing columns
• Creating indexes
• Updating constraints
• Inserting seed data

Liquibase will then apply these changes to the database in a controlled and repeatable way. When the application starts, Liquibase checks which migrations have already been applied and only runs any new changes.

This gives us safer and more consistent database changes across different environments with the addition of version-control.

Flyway is also a good alternative to Liquibase.

Implementing Liquibase With Spring Boot

Now we have some understanding of what Liquibase is and why we might use it, let's get stuck in!

Removing the Old Database

Before we implement Liquibase, we need to remove the old Postgres database that we've been using for development. If we don't do this, we would end up with conflict errors due to the users table already existing.

To do this, you can stop and remove the Postgres Docker container:

docker stop postgres && docker rm postgres

Then prune any unused Docker volumes:

docker volume prune

NOTE: this will remove all unused Docker volumes.

To be more surgical with volume removal, you can do:

docker volume ls

To list all Docker volumes, then:

docker volume rm <volume-name>

Replacing <volume-name> with the Postgres volume to remove.

Add Dependencies

Now we've remove the old database, let's start by adding a dependency for Liquibase to our build.gradle file:

// build.gradle
dependencies {
    //other deps...
    implementation 'org.liquibase:liquibase-core'
    // other deps...
}

We should also deactivate the ddl-auto like so to prevent Hibernate from updating our database schema:

spring:
  #...other properties
  jpa:
    hibernate:
      ddl-auto: none

NOTE: you may want to use the validate option in local development. However, using this in production can cause issue when rolling back changes.

Create a Changelog File

Then we need to create the following file:

src/main/resources/db/changelog/change-log.yaml

and add our change set for our users table:

databaseChangeLog:
  - changeSet:
      id: 1-create-users-table
      author: full-bearded-dev
      changes:
        - createTable:
            tableName: users
            columns:
              - column:
                  name: id
                  type: bigint
                  autoIncrement: true
                  constraints:
                    primaryKey: true
                    primaryKeyName: pk_users
                    nullable: false
              - column:
                  name: name
                  type: varchar(255)
                  constraints:
                    nullable: false
              - column:
                  name: email
                  type: varchar(255)
                  constraints:
                    nullable: false
                    unique: true
                    uniqueConstraintName: uk_users_email
              - column:
                  name: age
                  type: int
                  constraints:
                    nullable: false
              - column:
                  name: password
                  type: varchar(255)
                  constraints:
                    nullable: false
              - column:
                  name: role
                  type: varchar(50)
                  constraints:
                    nullable: false

Update Properties

We also need to tell our Spring Boot application where to find the new changelog file in our properties file:

spring:
  #...other properties
  liquibase:
    change-log: "classpath:/db/changelog/change-log.yaml"

Update the User Entity

We will also need to add the following IDENTITY generation type to our User.java entity:

//...
public class User {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name = "id")
    private Long id;
    //...other fields
}

This prevents us from receiving an error from using the default AUTO option which expects there to be a users_seq table already in our database. Instead IDENTITY will just auto-increment user IDs by 1, each time a new user is created.

Now when the application starts up, Liquibase will check the migration history and apply any changes that haven't been made. These changes are then stored in the database.

You should see something like this in the application logs when running the application for the first time:

INFO 93997 --- [crud.app] [           main] liquibase.ui                             : Running Changeset: db/changelog/change-log.yaml::1-create-users-table::full-bearded-dev
INFO 93997 --- [crud.app] [           main] liquibase.changelog                      : Table users created
INFO 93997 --- [crud.app] [           main] liquibase.changelog                      : ChangeSet db/changelog/change-log.yaml::1-create-users-table::full-bearded-dev ran successfully in 8ms

Liquibase Tracking Tables

Liquibase will also automatically create two important tables: DATABASECHANGELOG and DATABASECHANGELOGLOCK

DATABASECHANGELOG tracks which migrations ran, when they ran, who created them and checksums for validation. This prevents migrations from executing multiple times.

DATABASECHANGELOGLOCK prevents multiple application instances from applying migrations simultaneously.

Without locking, two application instances starting at the same time could both attempt to modify the schema.

Liquibase acquires a lock before running migrations and releases it afterwards. This is especially important in distributed deployments using technologies like Kubernetes.

If we wanted to, we can actually view and inspect these tables like so.

Run the following to exec into the Postgres database:

docker exec -it postgres psql -U dev -d test_db

Run \dt to display all tables, you should see something like this:

               List of relations
 Schema |         Name          | Type  | Owner 
--------+-----------------------+-------+-------
 public | databasechangelog     | table | dev
 public | databasechangeloglock | table | dev
 public | users                 | table | dev
(3 rows)

We can then run the following to see each table:

select * from databasechangelog;

select * from databasechangeloglock;

Handling Schema Failure Example

Now we've added our Liquibase changelog, lets see why database migration matters and how it can help us to catch issues quickly.

Let's say we add a display name to our User entity like so:

//...
public class User {
    
    //...other fields
    @Column(name = "display_name")
    private String displayName;
    //...other fields
}

But forget to add the field to the database changelog. When we start our application, we will get the following exception:

Caused by: org.postgresql.util.PSQLException: ERROR: column u1_0.display_name does not exist

This is actually a good thing as our application fails fast instead of running against an invalid schema.

To fix this, we just need to add a new database migration in our changelog:

databaseChangeLog:
#...1-create-users-table change set
  - changeSet:
      id: 2-add-display-name
      author: full-bearded-dev
      changes:
        - addColumn:
            tableName: users
            columns:
              - column:
                  name: display_name
                  type: varchar(255)

Best Practices

Here are some of the best practices I try to follow when working with Liquibase:

• Never edit old migrations after deployment, keep the data safe
• Create new migrations for new changes
• Keep migrations small and focused
• Use descriptive IDs
• Review generated SQL
• Test migrations locally before production
• Use validate instead of update in local development

What's Next?

So to recap we learnt:

• What data migrations are
• What is Hibernate DDL auto and how is it used to update our Database
• What is Liquibase and how is it used to update our Database
• When to use Hibernate DDL auto and when to use Liquibase
• How to implement Liquibase in Spring Boot
• How does Liquibase deal with invalid database schemas
• Some best practices for using Liquibase

GitHub Example

Stay tuned for the upcoming blog in the series about dockerizing our app and getting one step closer to deploying our application!