14-Migration-System.md 44 KB
Migration System Architecture
Document Status
- Created: 2026-03-13
- Status: Draft
- Author: Architecture design session
Overview
This document describes the architecture for a comprehensive migration system that supports application-defined migrations for the Implexus database. The system provides ordered execution, transaction safety, and migration history tracking.
Requirements
The migration system must:
- Handle initial setup routine - Support an initial setup/bootstrap migration
- Single transaction execution - Run migration logic within a single transaction
- Track migration history - Keep track of which migrations have been run
- Migration interface - Provide
up()anddown()functions for each migration - Ordered execution - Run pending migrations in the correct order
Architecture Overview
graph TB
subgraph Application Layer
AM[Application Migrations<br/>User-defined classes]
end
subgraph Migration System
MI[Migration Interface<br/>up/down methods]
MR[MigrationRunner<br/>Discovery and execution]
MS[MigrationStorage<br/>History tracking]
ME[MigrationError<br/>Error domain]
end
subgraph Engine Layer
EE[EmbeddedEngine]
TX[Transaction Support]
end
subgraph Storage Layer
DBM[Dbm Interface]
end
AM --> MI
MI --> MR
MR --> MS
MR --> TX
MS --> DBM
TX --> DBM
EE --> MR
EE --> MS
Components
1. Migration Interface
The Migration interface defines the contract that all migrations must implement.
File: src/Migrations/Migration.vala
namespace Implexus.Migrations {
/**
* Interface for database migrations.
*
* Migrations are application-defined classes that modify the database
* schema or data in a controlled, versioned manner.
*
* Each migration must implement:
* - A unique identifier (version)
* - An up() method to apply the migration
* - A down() method to reverse the migration (optional for irreversible migrations)
*
* Example:
* {{{
* public class CreateUsersTable : Object, Migration {
* public string version { get { return "2026031301"; } }
* public string description { get { return "Create users container"; } }
*
* public void up(Core.Engine engine) throws MigrationError {
* var root = engine.get_root();
* try {
* root.create_container("users");
* } catch (Core.EngineError e) {
* throw new MigrationError.EXECUTION_FAILED(
* "Failed to create users container: %s".printf(e.message)
* );
* }
* }
*
* public void down(Core.Engine engine) throws MigrationError {
* var root = engine.get_root();
* try {
* var users = root.get_child("users");
* if (users != null) {
* users.delete();
* }
* } catch (Core.EngineError e) {
* throw new MigrationError.EXECUTION_FAILED(
* "Failed to delete users container: %s".printf(e.message)
* );
* }
* }
* }
* }}}
*/
public interface Migration : Object {
/**
* Unique identifier for this migration.
*
* Best practice: Use a timestamp format like YYYYMMDDNN where NN is a
* sequence number within the day. This ensures proper ordering.
*
* Examples: "2026031301", "2026031302", "2026031501"
*/
public abstract string version { owned get; }
/**
* Human-readable description of what this migration does.
*/
public abstract string description { owned get; }
/**
* Applies the migration.
*
* This method is called when migrating forward. All operations
* are executed within a single transaction.
*
* @param engine The database engine to operate on
* @throws MigrationError if the migration fails
*/
public abstract void up(Core.Engine engine) throws MigrationError;
/**
* Reverses the migration.
*
* This method is called when rolling back. All operations
* are executed within a single transaction.
*
* Implementations may throw MigrationError.IRREVERSIBLE if the
* migration cannot be safely reversed.
*
* @param engine The database engine to operate on
* @throws MigrationError if the rollback fails
*/
public abstract void down(Core.Engine engine) throws MigrationError;
}
} // namespace Implexus.Migrations
2. MigrationRunner
The MigrationRunner discovers and executes migrations in the correct order.
File: src/Migrations/MigrationRunner.vala
namespace Implexus.Migrations {
/**
* Delegate for migration progress notifications.
*/
public delegate void MigrationProgressDelegate(Migration migration, bool is_up);
/**
* Discovers and executes migrations in version order.
*
* The MigrationRunner is responsible for:
* - Registering available migrations
* - Determining which migrations need to run
* - Executing migrations in the correct order within transactions
* - Recording migration history
*
* Example usage:
* {{{
* var runner = new MigrationRunner(engine);
*
* // Register migrations
* runner.register_migration(new CreateUsersTable());
* runner.register_migration(new AddEmailIndex());
* runner.register_migration(new SeedInitialData());
*
* // Run all pending migrations
* try {
* int count = runner.run_pending();
* print("Ran %d migrations\n", count);
* } catch (MigrationError e) {
* stderr.printf("Migration failed: %s\n", e.message);
* }
* }}}
*/
public class MigrationRunner : Object {
// === Private Fields ===
private weak Core.Engine _engine;
private MigrationStorage _storage;
private Invercargill.DataStructures.Dictionary<string, Migration> _migrations;
// === Constructors ===
/**
* Creates a new MigrationRunner for the given engine.
*
* @param engine The database engine to migrate
*/
public MigrationRunner(Core.Engine engine) {
_engine = engine;
_storage = new MigrationStorage(engine);
_migrations = new Invercargill.DataStructures.Dictionary<string, Migration>();
}
// === Migration Registration ===
/**
* Registers a migration to be available for execution.
*
* Migrations must be registered before calling run_pending() or rollback_to().
*
* @param migration The migration to register
* @throws MigrationError.VERSION_CONFLICT if a migration with the same version exists
*/
public void register_migration(Migration migration) throws MigrationError {
var version = migration.version;
if (_migrations.has(version)) {
throw new MigrationError.VERSION_CONFLICT(
"Migration version %s is already registered".printf(version)
);
}
_migrations.set(version, migration);
}
/**
* Registers multiple migrations at once.
*
* @param migrations Array of migrations to register
* @throws MigrationError if any registration fails
*/
public void register_migrations(Migration[] migrations) throws MigrationError {
foreach (var migration in migrations) {
register_migration(migration);
}
}
// === Status Queries ===
/**
* Gets all migration versions that have been applied.
*
* @return Sorted list of applied migration versions
*/
public Invercargill.Enumerable<string> get_applied_versions() {
return _storage.get_applied_versions();
}
/**
* Gets all migration versions that are registered but not yet applied.
*
* @return Sorted list of pending migration versions
*/
public Invercargill.Enumerable<string> get_pending_versions() {
var pending = new Invercargill.DataStructures.Vector<string>();
var applied = _storage.get_applied_set();
// Get all registered versions and sort them
var all_versions = new Invercargill.DataStructures.Vector<string>();
foreach (var version in _migrations.keys) {
all_versions.add(version);
}
all_versions.sort((a, b) => a.compare(b));
// Filter to only pending
foreach (var version in all_versions) {
if (!applied.contains(version)) {
pending.add(version);
}
}
return pending.as_enumerable();
}
/**
* Checks if a specific migration version has been applied.
*
* @param version The migration version to check
* @return true if the migration has been applied
*/
public bool is_applied(string version) {
return _storage.is_applied(version);
}
/**
* Gets the count of pending migrations.
*
* @return Number of migrations waiting to be applied
*/
public int get_pending_count() {
int count = 0;
var applied = _storage.get_applied_set();
foreach (var version in _migrations.keys) {
if (!applied.contains(version)) {
count++;
}
}
return count;
}
// === Execution ===
/**
* Runs all pending migrations in version order.
*
* Each migration runs in its own transaction. If a migration fails,
* the process stops and the failing migration is rolled back.
*
* @param progress Optional callback for progress notifications
* @return Number of migrations that were run
* @throws MigrationError if any migration fails
*/
public int run_pending(MigrationProgressDelegate? progress = null) throws MigrationError {
var pending = get_pending_versions();
int count = 0;
foreach (var version in pending) {
var migration = _migrations.get(version);
if (migration == null) {
throw new MigrationError.NOT_FOUND(
"Migration %s not found".printf(version)
);
}
run_single((!) migration, true, progress);
count++;
}
return count;
}
/**
* Runs a specific migration.
*
* @param version The version of the migration to run
* @param progress Optional callback for progress notification
* @throws MigrationError if the migration fails or is already applied
*/
public void run_one(string version, MigrationProgressDelegate? progress = null) throws MigrationError {
if (_storage.is_applied(version)) {
throw new MigrationError.ALREADY_APPLIED(
"Migration %s is already applied".printf(version)
);
}
var migration = _migrations.get(version);
if (migration == null) {
throw new MigrationError.NOT_FOUND(
"Migration %s not found".printf(version)
);
}
run_single((!) migration, true, progress);
}
/**
* Rolls back to a specific version.
*
* Runs the down() method of all migrations after the target version,
* in reverse order.
*
* @param target_version The version to roll back to (exclusive - this version remains applied)
* @param progress Optional callback for progress notifications
* @return Number of migrations that were rolled back
* @throws MigrationError if any rollback fails
*/
public int rollback_to(string target_version, MigrationProgressDelegate? progress = null) throws MigrationError {
var applied = _storage.get_applied_versions();
int count = 0;
// Roll back in reverse order
var to_rollback = new Invercargill.DataStructures.Vector<string>();
foreach (var version in applied) {
if (version.compare(target_version) > 0) {
to_rollback.add(version);
}
}
// Sort in reverse order
to_rollback.sort((a, b) => b.compare(a));
foreach (var version in to_rollback) {
var migration = _migrations.get(version);
if (migration == null) {
throw new MigrationError.NOT_FOUND(
"Migration %s not found for rollback".printf(version)
);
}
run_single((!) migration, false, progress);
count++;
}
return count;
}
/**
* Rolls back the most recently applied migration.
*
* @param progress Optional callback for progress notification
* @throws MigrationError if no migrations are applied or rollback fails
*/
public void rollback_last(MigrationProgressDelegate? progress = null) throws MigrationError {
var applied = _storage.get_applied_versions();
string? last_version = null;
foreach (var version in applied) {
last_version = version; // Last one due to sorted order
}
if (last_version == null) {
throw new MigrationError.NO_MIGRATIONS(
"No migrations have been applied"
);
}
var migration = _migrations.get((!) last_version);
if (migration == null) {
throw new MigrationError.NOT_FOUND(
"Migration %s not found for rollback".printf((!) last_version)
);
}
run_single((!) migration, false, progress);
}
// === Internal Methods ===
/**
* Runs a single migration within a transaction.
*/
private void run_single(Migration migration, bool is_up, MigrationProgressDelegate? progress)
throws MigrationError {
var engine = _engine;
if (engine == null) {
throw new MigrationError.ENGINE_ERROR("Engine reference is invalid");
}
// Execute within transaction
try {
var tx = ((!) engine).begin_transaction();
try {
if (is_up) {
migration.up((!) engine);
_storage.record_migration(migration.version, migration.description);
} else {
migration.down((!) engine);
_storage.remove_migration(migration.version);
}
tx.commit();
if (progress != null) {
progress(migration, is_up);
}
} catch (MigrationError e) {
tx.rollback();
throw e;
} catch (Core.EngineError e) {
tx.rollback();
throw new MigrationError.EXECUTION_FAILED(
"Migration %s failed: %s".printf(migration.version, e.message)
);
}
} catch (Core.EngineError e) {
throw new MigrationError.TRANSACTION_ERROR(
"Failed to begin transaction: %s".printf(e.message)
);
}
}
}
} // namespace Implexus.Migrations
3. MigrationStorage
The MigrationStorage class handles persistence of migration history using the existing Dbm infrastructure.
File: src/Migrations/MigrationStorage.vala
namespace Implexus.Migrations {
/**
* Stores and retrieves migration history.
*
* Migration history is persisted in the database using a dedicated
* key prefix. This ensures migration state survives application restarts.
*
* Key format: migration:<version>
* Value: Serialized (timestamp, description)
*/
public class MigrationStorage : Object {
// === Constants ===
private const string PREFIX = "migration:";
private const string INDEX_KEY = "migration:index";
// === Private Fields ===
private weak Core.Engine _engine;
private Storage.Dbm? _dbm;
// === Constructors ===
/**
* Creates a new MigrationStorage for the given engine.
*
* @param engine The database engine
*/
public MigrationStorage(Core.Engine engine) {
_engine = engine;
// Get Dbm from engine configuration if available
var config = engine.configuration;
var storage = config.storage;
// Try to get Dbm from BasicStorage
var basic_storage = (storage as Storage.BasicStorage);
if (basic_storage != null) {
_dbm = ((!) basic_storage).dbm;
}
}
// === Recording Migrations ===
/**
* Records that a migration has been applied.
*
* @param version The migration version
* @param description The migration description
* @throws MigrationError if the record cannot be saved
*/
public void record_migration(string version, string description) throws MigrationError {
if (_dbm == null) {
throw new MigrationError.STORAGE_ERROR("Database storage not available");
}
var dbm = (!) _dbm;
string key = PREFIX + version;
// Store migration record
var writer = new Storage.ElementWriter();
writer.write_element(new Invercargill.NativeElement<int64?>(new DateTime.now_utc().to_unix()));
writer.write_element(new Invercargill.NativeElement<string>(description));
try {
dbm.set(key, writer.to_binary_data());
} catch (Storage.StorageError e) {
throw new MigrationError.STORAGE_ERROR(
"Failed to record migration: %s".printf(e.message)
);
}
// Update index
update_index(dbm, version, true);
}
/**
* Removes a migration record (for rollbacks).
*
* @param version The migration version to remove
* @throws MigrationError if the record cannot be removed
*/
public void remove_migration(string version) throws MigrationError {
if (_dbm == null) {
throw new MigrationError.STORAGE_ERROR("Database storage not available");
}
var dbm = (!) _dbm;
string key = PREFIX + version;
try {
dbm.delete(key);
} catch (Storage.StorageError e) {
// Key may not exist, that's fine
}
// Update index
update_index(dbm, version, false);
}
// === Querying Migrations ===
/**
* Gets all applied migration versions in sorted order.
*
* @return Sorted enumerable of version strings
*/
public Invercargill.Enumerable<string> get_applied_versions() {
var versions = new Invercargill.DataStructures.Vector<string>();
if (_dbm == null) {
return versions.as_enumerable();
}
// Load from index
var index_data = ((!) _dbm).get(INDEX_KEY);
if (index_data == null) {
// Fall back to scanning keys
foreach (var key in ((!) _dbm).keys) {
if (key.has_prefix(PREFIX)) {
var version = key.substring(PREFIX.length);
versions.add(version);
}
}
versions.sort((a, b) => a.compare(b));
return versions.as_enumerable();
}
// Parse index
var reader = new Storage.ElementReader((!) index_data);
try {
var element = reader.read_element();
if (!element.is_null()) {
var array = element.as<Invercargill.Enumerable<Invercargill.Element>>();
foreach (var item in array) {
if (!item.is_null()) {
versions.add(item.as<string>());
}
}
}
} catch (Invercargill.ElementError e) {
// Fall back to empty list
}
return versions.as_enumerable();
}
/**
* Gets a set of applied migration versions for efficient lookup.
*
* @return Set of applied version strings
*/
public Invercargill.DataStructures.Set<string> get_applied_set() {
var set = new Invercargill.DataStructures.HashSet<string>();
foreach (var version in get_applied_versions()) {
set.add(version);
}
return set;
}
/**
* Checks if a specific migration has been applied.
*
* @param version The migration version to check
* @return true if the migration has been applied
*/
public bool is_applied(string version) {
if (_dbm == null) {
return false;
}
string key = PREFIX + version;
return ((!) _dbm).has_key(key);
}
/**
* Gets detailed information about an applied migration.
*
* @param version The migration version
* @return Migration record, or null if not applied
*/
public MigrationRecord? get_migration_record(string version) {
if (_dbm == null) {
return null;
}
string key = PREFIX + version;
var data = ((!) _dbm).get(key);
if (data == null) {
return null;
}
var reader = new Storage.ElementReader((!) data);
try {
var timestamp_element = reader.read_element();
var desc_element = reader.read_element();
int64? timestamp = timestamp_element.as<int64?>();
string description = desc_element.as<string>();
return new MigrationRecord(
version,
description,
timestamp != null ? new DateTime.from_unix_utc((!) timestamp) : null
);
} catch (Invercargill.ElementError e) {
return null;
}
}
// === Private Methods ===
/**
* Updates the migration index.
*/
private void update_index(Storage.Dbm dbm, string version, bool add) {
var versions = new Invercargill.DataStructures.Vector<string>();
// Load existing index
var index_data = dbm.get(INDEX_KEY);
if (index_data != null) {
var reader = new Storage.ElementReader((!) index_data);
try {
var element = reader.read_element();
if (!element.is_null()) {
var array = element.as<Invercargill.Enumerable<Invercargill.Element>>();
foreach (var item in array) {
if (!item.is_null()) {
string existing = item.as<string>();
if (!add || existing != version) {
versions.add(existing);
}
}
}
}
} catch (Invercargill.ElementError e) {
// Start fresh
}
}
if (add) {
versions.add(version);
}
// Sort
versions.sort((a, b) => a.compare(b));
// Save index
if (versions.length == 0) {
try {
dbm.delete(INDEX_KEY);
} catch (Storage.StorageError e) {
// Ignore
}
return;
}
var array = new Invercargill.DataStructures.Vector<Invercargill.Element>();
foreach (var v in versions) {
array.add(new Invercargill.NativeElement<string>(v));
}
var writer = new Storage.ElementWriter();
writer.write_element(new Invercargill.NativeElement<Invercargill.Enumerable<Invercargill.Element>>(array));
try {
dbm.set(INDEX_KEY, writer.to_binary_data());
} catch (Storage.StorageError e) {
// Ignore - not critical
}
}
}
/**
* Represents a record of an applied migration.
*/
public class MigrationRecord : Object {
/**
* The migration version.
*/
public string version { get; construct set; }
/**
* The migration description.
*/
public string description { get; construct set; }
/**
* When the migration was applied, or null if unknown.
*/
public DateTime? applied_at { get; construct set; }
/**
* Creates a new MigrationRecord.
*/
public MigrationRecord(string version, string description, DateTime? applied_at) {
Object(version: version, description: description, applied_at: applied_at);
}
}
} // namespace Implexus.Migrations
4. MigrationError
Error domain for migration-related errors.
File: src/Migrations/MigrationError.vala
namespace Implexus.Migrations {
/**
* Error domain for migration operations.
*/
public errordomain MigrationError {
/**
* The migration was not found.
*/
NOT_FOUND,
/**
* A migration with this version is already registered or applied.
*/
VERSION_CONFLICT,
/**
* The migration is already applied.
*/
ALREADY_APPLIED,
/**
* The migration cannot be reversed.
*/
IRREVERSIBLE,
/**
* The migration execution failed.
*/
EXECUTION_FAILED,
/**
* Transaction error during migration.
*/
TRANSACTION_ERROR,
/**
* Storage error during migration.
*/
STORAGE_ERROR,
/**
* Engine reference error.
*/
ENGINE_ERROR,
/**
* No migrations are available or applied.
*/
NO_MIGRATIONS
}
} // namespace Implexus.Migrations
5. Bootstrap Migration
A special migration for initial database setup.
File: src/Migrations/BootstrapMigration.vala
namespace Implexus.Migrations {
/**
* A bootstrap migration that runs before any other migrations.
*
* This migration is automatically applied when the database is first
* created. It can be extended by applications to perform initial setup
* such as creating required containers or seeding default data.
*
* The bootstrap migration always uses version "0000000000" to ensure
* it runs first.
*
* Example:
* {{{
* public class MyAppBootstrap : BootstrapMigration {
* public override void up(Core.Engine engine) throws MigrationError {
* base.up(engine);
*
* var root = engine.get_root();
* try {
* // Create application-specific containers
* root.create_container("config");
* root.create_container("sessions");
*
* // Seed initial data
* var config = root.get_child("config");
* var settings = config.create_document("settings", "AppConfig");
* settings.set_entity_property("version", new Invercargill.NativeElement<string>("1.0.0"));
* } catch (Core.EngineError e) {
* throw new MigrationError.EXECUTION_FAILED(
* "Bootstrap failed: %s".printf(e.message)
* );
* }
* }
* }
* }}}
*/
public class BootstrapMigration : Object, Migration {
/**
* Bootstrap version - always runs first.
*/
public string version { owned get { return "0000000000"; } }
/**
* Description of the bootstrap migration.
*/
public virtual string description { owned get { return "Initial database setup"; } }
/**
* Performs initial database setup.
*
* The base implementation ensures the root container exists.
* Override to add application-specific setup.
*/
public virtual void up(Core.Engine engine) throws MigrationError {
// Ensure root exists
try {
engine.get_root();
} catch (Core.EngineError e) {
throw new MigrationError.EXECUTION_FAILED(
"Failed to initialize root: %s".printf(e.message)
);
}
}
/**
* Bootstrap migration cannot be reversed.
*/
public void down(Core.Engine engine) throws MigrationError {
throw new MigrationError.IRREVERSIBLE(
"Bootstrap migration cannot be reversed"
);
}
}
} // namespace Implexus.Migrations
File Organization
New files should be placed in the following structure:
src/
├── Migration/
│ ├── meson.build
│ ├── Migration.vala # Migration interface
│ ├── MigrationRunner.vala # Migration execution engine
│ ├── MigrationStorage.vala # History persistence
│ ├── MigrationError.vala # Error domain
│ └── BootstrapMigration.vala # Initial setup migration
meson.build
migration_sources = files(
'Migration.vala',
'MigrationRunner.vala',
'MigrationStorage.vala',
'MigrationError.vala',
'BootstrapMigration.vala'
)
The migration sources should be added to the main library sources in src/meson.build.
Integration with Engine
Engine Interface Extension
Add migration support to the Engine interface:
// In src/Core/Engine.vala
public interface Engine : Object {
// ... existing methods ...
/**
* Creates a new MigrationRunner for this engine.
*
* @return A new MigrationRunner instance
*/
public abstract Migration.MigrationRunner create_migration_runner();
}
EmbeddedEngine Implementation
// In src/Engine/EmbeddedEngine.vala
public class EmbeddedEngine : Object, Core.Engine {
// ... existing code ...
/**
* {@inheritDoc}
*/
public Migration.MigrationRunner create_migration_runner() {
return new Migration.MigrationRunner(this);
}
}
API Examples
Defining Migrations
// Migration 1: Create users container
public class CreateUsersContainer : Object, Implexus.Migrations.Migration {
public string version { owned get { return "2026031301"; } }
public string description { owned get { return "Create users container"; } }
public void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
try {
var root = engine.get_root();
root.create_container("users");
} catch (Implexus.Core.EngineError e) {
throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
"Failed to create users: %s".printf(e.message)
);
}
}
public void down(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
try {
var root = engine.get_root();
var users = root.get_child("users");
if (users != null) {
users.delete();
}
} catch (Implexus.Core.EngineError e) {
throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
"Failed to remove users: %s".printf(e.message)
);
}
}
}
// Migration 2: Add email index
public class AddEmailIndex : Object, Implexus.Migrations.Migration {
public string version { owned get { return "2026031302"; } }
public string description { owned get { return "Add email index for users"; } }
public void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
try {
var root = engine.get_root();
var users = (Implexus.Entities.Container?) root.get_child("users");
if (users != null) {
((!) users).create_index("by_email", "User", "email");
}
} catch (Implexus.Core.EngineError e) {
throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
"Failed to create email index: %s".printf(e.message)
);
}
}
public void down(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
try {
var root = engine.get_root();
var users = root.get_child("users");
if (users != null) {
var index = ((!) users).get_child("by_email");
if (index != null) {
index.delete();
}
}
} catch (Implexus.Core.EngineError e) {
throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
"Failed to remove email index: %s".printf(e.message)
);
}
}
}
// Migration 3: Seed initial data (irreversible)
public class SeedInitialData : Object, Implexus.Migrations.Migration {
public string version { owned get { return "2026031303"; } }
public string description { owned get { return "Seed initial configuration data"; } }
public void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
try {
var root = engine.get_root();
var config = root.create_container("config");
var settings = config.create_document("settings", "AppSettings");
settings.set_entity_property("theme", new Invercargill.NativeElement<string>("dark"));
settings.set_entity_property("max_users", new Invercargill.NativeElement<int>(100));
} catch (Implexus.Core.EngineError e) {
throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
"Failed to seed data: %s".printf(e.message)
);
}
}
public void down(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
// This migration is irreversible - data loss is not acceptable
throw new Implexus.Migrations.MigrationError.IRREVERSIBLE(
"Seed data migration cannot be reversed"
);
}
}
Running Migrations
// Basic migration run
public void run_migrations(Implexus.Core.Engine engine) {
var runner = engine.create_migration_runner();
try {
// Register all migrations
runner.register_migration(new CreateUsersContainer());
runner.register_migration(new AddEmailIndex());
runner.register_migration(new SeedInitialData());
// Run pending migrations
int count = runner.run_pending();
print("Successfully ran %d migrations\n", count);
} catch (Implexus.Migrations.MigrationError e) {
stderr.printf("Migration failed: %s\n", e.message);
}
}
// With progress reporting
public void run_migrations_with_progress(Implexus.Core.Engine engine) {
var runner = engine.create_migration_runner();
try {
runner.register_migrations(new Implexus.Migrations.Migration[] {
new CreateUsersContainer(),
new AddEmailIndex(),
new SeedInitialData()
});
// Run with progress callback
int count = runner.run_pending((migration, is_up) => {
string direction = is_up ? "Up" : "Down";
print("[%s] %s: %s\n", direction, migration.version, migration.description);
});
print("Completed %d migrations\n", count);
} catch (Implexus.Migrations.MigrationError e) {
stderr.printf("Migration failed: %s\n", e.message);
}
}
// Check status before running
public void check_and_run_migrations(Implexus.Core.Engine engine) {
var runner = engine.create_migration_runner();
try {
runner.register_migrations(new Implexus.Migrations.Migration[] {
new CreateUsersContainer(),
new AddEmailIndex(),
new SeedInitialData()
});
// Check status
int pending = runner.get_pending_count();
if (pending == 0) {
print("Database is up to date\n");
return;
}
print("There are %d pending migrations:\n", pending);
foreach (var version in runner.get_pending_versions()) {
print(" - %s\n", version);
}
// Run them
runner.run_pending();
} catch (Implexus.Migrations.MigrationError e) {
stderr.printf("Migration failed: %s\n", e.message);
}
}
Rolling Back Migrations
// Roll back last migration
public void rollback_last_migration(Implexus.Core.Engine engine) {
var runner = engine.create_migration_runner();
try {
runner.register_migrations(new Implexus.Migrations.Migration[] {
new CreateUsersContainer(),
new AddEmailIndex(),
new SeedInitialData()
});
runner.rollback_last((migration, is_up) => {
print("Rolled back: %s\n", migration.version);
});
} catch (Implexus.Migrations.MigrationError e) {
if (e is Implexus.Migrations.MigrationError.IRREVERSIBLE) {
stderr.printf("Cannot roll back irreversible migration: %s\n", e.message);
} else {
stderr.printf("Rollback failed: %s\n", e.message);
}
}
}
// Roll back to specific version
public void rollback_to_version(Implexus.Core.Engine engine, string target_version) {
var runner = engine.create_migration_runner();
try {
runner.register_migrations(new Implexus.Migrations.Migration[] {
new CreateUsersContainer(),
new AddEmailIndex(),
new SeedInitialData()
});
int count = runner.rollback_to(target_version);
print("Rolled back %d migrations\n", count);
} catch (Implexus.Migrations.MigrationError e) {
stderr.printf("Rollback failed: %s\n", e.message);
}
}
Using Bootstrap Migration
// Custom bootstrap for application
public class MyAppBootstrap : Implexus.Migrations.BootstrapMigration {
public override string description { owned get { return "MyApp initial setup"; } }
public override void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
// Call base to ensure root exists
base.up(engine);
try {
var root = engine.get_root();
// Create application structure
root.create_container("users");
root.create_container("projects");
root.create_container("settings");
// Create default admin user
var users = (Implexus.Entities.Container) root.get_child("users");
var admin = users.create_document("admin", "User");
admin.set_entity_property("role", new Invercargill.NativeElement<string>("admin"));
admin.set_entity_property("email", new Invercargill.NativeElement<string>("admin@example.com"));
} catch (Implexus.Core.EngineError e) {
throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
"Bootstrap failed: %s".printf(e.message)
);
}
}
}
// Using bootstrap with regular migrations
public void initialize_database(Implexus.Core.Engine engine) {
var runner = engine.create_migration_runner();
try {
// Register bootstrap first
runner.register_migration(new MyAppBootstrap());
// Then register regular migrations
runner.register_migration(new CreateUsersContainer());
runner.register_migration(new AddEmailIndex());
// Run all pending (bootstrap will run first due to version ordering)
runner.run_pending();
} catch (Implexus.Migrations.MigrationError e) {
stderr.printf("Database initialization failed: %s\n", e.message);
}
}
Error Handling
Error Handling Strategy
Transaction Safety: Each migration runs in its own transaction. If a migration fails, only that migration is rolled back.
Atomic Operations: Within a migration, all operations are atomic. Either all changes apply, or none do.
Irreversible Migrations: Migrations that cannot be safely reversed should throw
MigrationError.IRREVERSIBLEin theirdown()method.Progress Preservation: Successfully applied migrations are recorded even if later migrations fail. The system can be resumed after fixing the failing migration.
Error Handling Flow
sequenceDiagram
participant App as Application
participant Runner as MigrationRunner
participant TX as Transaction
participant Storage as MigrationStorage
participant DB as Database
App->>Runner: run_pending()
loop For each pending migration
Runner->>TX: begin_transaction()
Runner->>App: migration.up()
alt Success
App->>DB: Apply changes
Runner->>Storage: record_migration()
Storage->>DB: Save record
Runner->>TX: commit()
Runner->>App: progress callback
else Failure
App-->>Runner: throws EngineError
Runner->>TX: rollback()
Runner-->>App: throws MigrationError
end
end
Error Recovery
public void run_with_recovery(Implexus.Core.Engine engine) {
var runner = engine.create_migration_runner();
try {
runner.register_migrations(get_all_migrations());
runner.run_pending();
} catch (Implexus.Migrations.MigrationError e) {
// Check what was applied before failure
var applied = runner.get_applied_versions();
var pending = runner.get_pending_versions();
stderr.printf("Migration failed: %s\n", e.message);
stderr.printf("Applied: %d, Pending: %d\n",
applied.length, pending.length);
// Log for manual recovery
log_migration_failure(e, applied, pending);
// Depending on error type, may be able to retry
if (e is Implexus.Migrations.MigrationError.TRANSACTION_ERROR) {
stderr.printf("Transaction error - may be retried\n");
} else if (e is Implexus.Migrations.MigrationError.EXECUTION_FAILED) {
stderr.printf("Execution failed - check migration code\n");
}
}
}
Storage Schema
Key Schema
| Key Pattern | Description |
|---|---|
migration:<version> |
Record of an applied migration |
migration:index |
Sorted list of all applied migration versions |
Data Format
migration:
Element[] {
int64 timestamp, // Unix timestamp when applied
string description // Migration description
}
migration:index
Element[] {
string[] versions // Sorted array of applied version strings
}
Design Decisions
Version Format: Using string-based versions (e.g., "2026031301") instead of integers allows for:
- Natural ordering by timestamp
- Human-readable identification
- No collision issues in distributed development
One Transaction Per Migration: Each migration runs in its own transaction rather than batching all migrations in one transaction. This:
- Allows partial progress recovery
- Prevents long-running transactions
- Matches common migration tool patterns
Separate MigrationStorage: Storage logic is separated from the runner to:
- Allow alternative storage implementations
- Enable testing with mock storage
- Follow single responsibility principle
Bootstrap as Regular Migration: The bootstrap migration is implemented as a regular migration with a special version number rather than a separate mechanism. This:
- Ensures it's tracked in history
- Allows rollback (if reversible)
- Uses the same infrastructure
No Auto-Discovery: Migrations must be explicitly registered rather than auto-discovered. This:
- Gives applications full control
- Avoids reflection/metadata complexity
- Works well with Vala's compilation model
Future Considerations
Migration Validation: Could add a
validate()method to migrations for pre-flight checks.Dry Run Mode: Could add a
--dry-runoption that shows what would happen without executing.Migration Dependencies: Could add support for migrations that depend on other migrations.
Conditional Migrations: Could add support for migrations that only run under certain conditions.
Migration Groups: Could add support for grouping migrations into named sets that can be applied together.