Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions docs/documentation/upgrading/topics/migrate_db.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,14 @@ kc.[sh|bat] start --spi-connections-jpa--quarkus--migration-export=<path>/<file.

After the changes have been written to the file, the server exits.

For an empty database the script also creates the Liquibase `DATABASECHANGELOG` table, which it records the applied
changesets in. It does not create the `DATABASECHANGELOGLOCK` table: {project_name} creates that one itself on
startup, so it is missing from a database that was only ever set up by running the script.

Keep this in mind when the database user is not allowed to create tables. Even with the manual strategy,
{project_name} creates both Liquibase bookkeeping tables, `DATABASECHANGELOG` and `DATABASECHANGELOGLOCK`, on the
database it is started against. Have an administrator create them up front, both on the database used to generate
the script and on the database the script is applied to. When they already exist, the generated script contains no
`CREATE TABLE` for them, so it can also be applied to the database it was generated from.
For an empty database the script also creates the two Liquibase bookkeeping tables, `DATABASECHANGELOG`, which it
records the applied changesets in, and `DATABASECHANGELOGLOCK`. A database that was set up purely by running the
script can therefore be started against by a database user that is not allowed to create tables.

Generating the script is different: even with the manual strategy, {project_name} creates both bookkeeping tables
itself on the database it generates the script against. Use a database user that is allowed to create tables there,
or have an administrator create the two tables up front. When they already exist before the server starts, the
generated script contains no `CREATE TABLE` for them, so it can also be applied to the database it was generated from.

For further details on how to run the statements in the file against the database, see the documentation for your relational database.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
import liquibase.snapshot.SnapshotGeneratorFactory;
import liquibase.statement.SqlStatement;
import liquibase.statement.core.AddColumnStatement;
import liquibase.statement.core.CreateDatabaseChangeLogLockTableStatement;
import liquibase.statement.core.CreateDatabaseChangeLogTableStatement;
import liquibase.statement.core.SetNullableStatement;
import liquibase.statement.core.UpdateStatement;
Expand All @@ -77,6 +78,9 @@ public class LiquibaseJpaUpdaterProvider implements JpaUpdaterProvider {
// Did the master changelog table already exist when this server started, before validation created it?
// Only the master table is relevant: Liquibase still writes the DDL of custom provider changelog tables itself.
private Boolean masterChangelogTablePreExisted;
// Did the lock table already exist when this server started? Recorded during validation, which runs before
// the database lock is taken and CustomLockService creates the table.
private Boolean lockTablePreExisted;

public LiquibaseJpaUpdaterProvider(KeycloakSession session) {
this.session = session;
Expand Down Expand Up @@ -146,6 +150,10 @@ protected void updateChangeSet(KeycloakLiquibase liquibase, Writer exportWriter)
if (masterChangelogTablePreExisted == null && isMasterChangelogTable(database)) {
masterChangelogTablePreExisted = changelogTable != null;
}
// The same is not possible for the lock table: by now the database lock is held, so CustomLockService has
// already created it and it always exists here. Validation is the only point at which its state before this
// server started can be observed, and both connection provider factories validate before they export. If it
// was not observed, the DDL is left out rather than guessed, so that the script can never fail on it.

if (changelogTable != null) {
boolean hasDeploymentIdColumn = changelogTable.getColumn(DEPLOYMENT_ID_COLUMN) != null;
Expand Down Expand Up @@ -228,15 +236,26 @@ private void outputChangeLogTableCreationScript(Liquibase liquibase, final Write
// instead, but only when the table was absent before this server started: an administrator may have
// pre-created it so that a user without DDL rights could produce the script at all, and in that case the
// target database already has the table and the DDL would fail the script with "already exists".
// DatabaseChangeLogLockTable is created before this code is executed and recreated if it does not exist automatically
// in org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService.init() called indirectly from
// KeycloakApplication constructor (search for waitForLock() call). Hence it is not included in the creation script.
if (isMasterChangelogTable(database) && Boolean.FALSE.equals(masterChangelogTablePreExisted)) {
// Same comment Liquibase emits when it writes this statement itself
loggingExecutor.comment("Create Database Change Log Table");
loggingExecutor.execute(new CreateDatabaseChangeLogTableStatement());
}

// The lock table is created by CustomLockService.init() before this code runs, when the database lock is
// taken, so it is always present by now and Liquibase never writes its DDL either. It was left out of the
// script on the assumption that the script is applied to the database it was generated from, where it
// therefore already exists. That does not hold when the script is generated against one database and
// applied to an empty one, which is the only way to obtain a script at all for a database user without
// DDL privileges: such a database would be left without the lock table and the first startup would fail
// trying to create it. Emit it under the same rule as the changelog table, and only once, with the master
// changelog - custom provider changelogs share the same lock table.
// Boolean.FALSE: a null (not observed, see updateChangeSet) deliberately emits nothing.
if (isMasterChangelogTable(database) && Boolean.FALSE.equals(lockTablePreExisted)) {
loggingExecutor.comment("Create Database Lock Table");
loggingExecutor.execute(new CreateDatabaseChangeLogLockTableStatement());
}

// For MySQL, add primary key to DATABASECHANGELOG table (handled by MySQLCustomChangeLogHistoryService at runtime)
ChangeLogHistoryService changeLogHistoryService = ChangeLogHistoryServiceFactory.getInstance().getChangeLogService(database);
if (changeLogHistoryService instanceof MySQLCustomChangeLogHistoryService customChangeLogHistoryService) {
Expand Down Expand Up @@ -325,6 +344,9 @@ private void recordWhetherMasterChangelogTablePreExisted(Database database) thro
.getDatabaseChangeLogTable(new SnapshotControl(database, false, Table.class, Column.class), database);
masterChangelogTablePreExisted = table != null;
}
if (lockTablePreExisted == null && isMasterChangelogTable(database)) {
lockTablePreExisted = SnapshotGeneratorFactory.getInstance().hasDatabaseChangeLogLockTable(database);
}
Comment thread
Copilot marked this conversation as resolved.
Resolved
}

private static boolean isMasterChangelogTable(Database database) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
import liquibase.snapshot.SnapshotGeneratorFactory;
import liquibase.statement.SqlStatement;
import liquibase.statement.core.AddColumnStatement;
import liquibase.statement.core.CreateDatabaseChangeLogLockTableStatement;
import liquibase.statement.core.CreateDatabaseChangeLogTableStatement;
import liquibase.statement.core.SetNullableStatement;
import liquibase.statement.core.UpdateStatement;
Expand All @@ -78,6 +79,9 @@ public class QuarkusJpaUpdaterProvider implements JpaUpdaterProvider {
// Did the master changelog table already exist when this server started, before validation created it?
// Only the master table is relevant: Liquibase still writes the DDL of custom provider changelog tables itself.
private Boolean masterChangelogTablePreExisted;
// Did the lock table already exist when this server started? Recorded during validation, which runs before
// the database lock is taken and CustomLockService creates the table.
private Boolean lockTablePreExisted;

public QuarkusJpaUpdaterProvider(KeycloakSession session) {
this.session = session;
Expand Down Expand Up @@ -149,6 +153,10 @@ protected void updateChangeSet(KeycloakLiquibase liquibase, Writer exportWriter)
if (masterChangelogTablePreExisted == null && isMasterChangelogTable(database)) {
masterChangelogTablePreExisted = changelogTable != null;
}
// The same is not possible for the lock table: by now the database lock is held, so CustomLockService has
// already created it and it always exists here. Validation is the only point at which its state before this
// server started can be observed, and both connection provider factories validate before they export. If it
// was not observed, the DDL is left out rather than guessed, so that the script can never fail on it.

if (changelogTable != null) {
boolean hasDeploymentIdColumn = changelogTable.getColumn(DEPLOYMENT_ID_COLUMN) != null;
Expand Down Expand Up @@ -231,15 +239,26 @@ private void outputChangeLogTableCreationScript(Liquibase liquibase, final Write
// instead, but only when the table was absent before this server started: an administrator may have
// pre-created it so that a user without DDL rights could produce the script at all, and in that case the
// target database already has the table and the DDL would fail the script with "already exists".
// DatabaseChangeLogLockTable is created before this code is executed and recreated if it does not exist automatically
// in org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService.init() called indirectly from
// KeycloakApplication constructor (search for waitForLock() call). Hence it is not included in the creation script.
if (isMasterChangelogTable(database) && Boolean.FALSE.equals(masterChangelogTablePreExisted)) {
// Same comment Liquibase emits when it writes this statement itself
loggingExecutor.comment("Create Database Change Log Table");
loggingExecutor.execute(new CreateDatabaseChangeLogTableStatement());
}

// The lock table is created by CustomLockService.init() before this code runs, when the database lock is
// taken, so it is always present by now and Liquibase never writes its DDL either. It was left out of the
// script on the assumption that the script is applied to the database it was generated from, where it
// therefore already exists. That does not hold when the script is generated against one database and
// applied to an empty one, which is the only way to obtain a script at all for a database user without
// DDL privileges: such a database would be left without the lock table and the first startup would fail
// trying to create it. Emit it under the same rule as the changelog table, and only once, with the master
// changelog - custom provider changelogs share the same lock table.
// Boolean.FALSE: a null (not observed, see updateChangeSet) deliberately emits nothing.
if (isMasterChangelogTable(database) && Boolean.FALSE.equals(lockTablePreExisted)) {
loggingExecutor.comment("Create Database Lock Table");
loggingExecutor.execute(new CreateDatabaseChangeLogLockTableStatement());
Comment thread
ahus1 marked this conversation as resolved.
Resolved
}

// For MySQL, add primary key to DATABASECHANGELOG table (handled by MySQLCustomChangeLogHistoryService at runtime)
ChangeLogHistoryService changeLogHistoryService = ChangeLogHistoryServiceFactory.getInstance().getChangeLogService(database);
if (changeLogHistoryService instanceof MySQLCustomChangeLogHistoryService customChangeLogHistoryService) {
Expand Down Expand Up @@ -331,6 +350,9 @@ private Table snapshotChangelogTable(Database database) throws LiquibaseExceptio
if (masterChangelogTablePreExisted == null && isMasterChangelogTable(database)) {
masterChangelogTablePreExisted = table != null;
}
if (lockTablePreExisted == null && isMasterChangelogTable(database)) {
lockTablePreExisted = SnapshotGeneratorFactory.getInstance().hasDatabaseChangeLogLockTable(database);
}
Comment thread
Copilot marked this conversation as resolved.
Resolved
return table;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,5 +77,13 @@ void testCustomProviderChangesetExportedInManualMode(KeycloakRunner runner, RawD
output.split(java.util.regex.Pattern.quote(createTable), -1).length - 1, is(1));
assertThat("custom changelog table must be created before it is inserted into",
output.indexOf(createTable), lessThan(output.indexOf(insertInto)));

// The lock table is shared by all changelogs and is emitted only together with the master changelog,
// which is already applied here. An export of just the custom provider changelog must therefore not emit
// it, or the script would fail on this database. The pre-existence guard itself is covered by
// ManualMigrationEmptyDatabaseDistTest#testEmptyDatabaseExportSkipsPreCreatedBookkeepingTables.
String createLockTable = "CREATE TABLE PUBLIC.DATABASECHANGELOGLOCK (";
assertThat("lock table must only be emitted with the master changelog",
output.split(java.util.regex.Pattern.quote(createLockTable), -1).length - 1, is(0));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ public class ManualMigrationEmptyDatabaseDistTest {

private static final String CREATE_CHANGELOG_TABLE = "CREATE TABLE PUBLIC.DATABASECHANGELOG (";
private static final String INSERT_INTO_CHANGELOG = "INSERT INTO PUBLIC.DATABASECHANGELOG (";
private static final String CREATE_LOCK_TABLE = "CREATE TABLE PUBLIC.DATABASECHANGELOGLOCK (";

/**
* The manual migration strategy exists for database users without DDL privileges. On an empty
Expand Down Expand Up @@ -82,12 +83,51 @@ void testEmptyDatabaseExportContainsChangelogTableDdl(KeycloakRunner runner, Raw
assertThat("changelog table must be created before it is inserted into",
output.indexOf(CREATE_CHANGELOG_TABLE), lessThan(output.indexOf(INSERT_INTO_CHANGELOG)));

// The server creates Liquibase's lock table itself on startup, which a database user without DDL privileges
// is not allowed to do. A database populated only by this script must therefore get it from the script.
assertThat("lock table must be created exactly once",
countOccurrences(output, CREATE_LOCK_TABLE), is(1));

// changeSetExecuted preconditions are resolved against the changelog table, so it must still exist while
// the script is generated. Were it missing, they would all resolve to "not executed" and changesets that
// should be marked as ran would be emitted instead - this one carries MySQL-only index syntax.
assertThat(output, not(containsString("VALUE(255)")));
}

/**
* An administrator may create both Liquibase bookkeeping tables up front, so that a database user without DDL
* privileges can produce the script at all. The script must then not create them again: it is meant to be
* applied to that same database, where the DDL would fail with "already exists". Producing the script once
* leaves exactly that state behind, as the server creates both tables on the live connection while exporting.
*/
@Test
void testEmptyDatabaseExportSkipsPreCreatedBookkeepingTables(KeycloakRunner runner, RawDistRootPath rawDistRootPath) throws IOException {
RawKeycloakDistribution rawDist = runner.getDistribution(RawKeycloakDistribution.class);
FileUtil.deleteDirectory(rawDist.getDistPath().resolve("data").resolve("h2").toAbsolutePath());

// First export on an empty database: creates the changelog and lock tables on the live connection
runner.run("start-dev",
"--spi-connections-jpa-quarkus-migration-strategy=manual",
"--spi-connections-jpa-quarkus-initialize-empty=false")
.assertMessage("Database not initialized, please initialize database with");

// Second export against the same, still empty database, where both tables now pre-exist
CLIResult result = runner.run("start-dev",
"--spi-connections-jpa-quarkus-migration-strategy=manual",
"--spi-connections-jpa-quarkus-initialize-empty=false");

result.assertMessage("Database not initialized, please initialize database with");

File script = rawDistRootPath.getDistRootPath().resolve("bin").resolve("keycloak-database-update.sql").toFile();
String output = FileUtils.readFileToString(script, Charset.defaultCharset());

assertThat(output, containsString(INSERT_INTO_CHANGELOG));
assertThat("changelog table must not be created when it already existed",
countOccurrences(output, CREATE_CHANGELOG_TABLE), is(0));
assertThat("lock table must not be created when it already existed",
countOccurrences(output, CREATE_LOCK_TABLE), is(0));
}

private static int countOccurrences(String haystack, String needle) {
return haystack.split(Pattern.quote(needle), -1).length - 1;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@ public String assertManualDbInitialization(CLIResult cliResult, RawDistRootPath
var count = countMatches(outputLowerCase, "create table public.databasechangelog (")
+ countMatches(outputLowerCase, "create table keycloak.databasechangelog (");
assertThat(count, is(1));

// ... and the lock table, which the server would otherwise have to create itself on first startup
var lockCount = countMatches(outputLowerCase, "create table public.databasechangeloglock (")
+ countMatches(outputLowerCase, "create table keycloak.databasechangeloglock (");
assertThat(lockCount, is(1));
assertThat(output, containsString("Change Log: META-INF/jpa-changelog-master.xml"));
assertThat(output, containsString("Changeset META-INF/jpa-changelog-26.2.6.xml"));

Expand Down
Loading