# Migration System Architecture

## Document Status
- **Created**: 2026-03-13
- **Status**: Draft
- **Author**: Architecture design session

## Overview

This document describes the architecture for a comprehensive migration system that supports application-defined migrations for the Implexus database. The system provides ordered execution, transaction safety, and migration history tracking.

## Requirements

The migration system must:
1. **Handle initial setup routine** - Support an initial setup/bootstrap migration
2. **Single transaction execution** - Run migration logic within a single transaction
3. **Track migration history** - Keep track of which migrations have been run
4. **Migration interface** - Provide `up()` and `down()` functions for each migration
5. **Ordered execution** - Run pending migrations in the correct order

## Architecture Overview

```mermaid
graph TB
    subgraph Application Layer
        AM[Application Migrations<br/>User-defined classes]
    end
    
    subgraph Migration System
        MI[Migration Interface<br/>up/down methods]
        MR[MigrationRunner<br/>Discovery and execution]
        MS[MigrationStorage<br/>History tracking]
        ME[MigrationError<br/>Error domain]
    end
    
    subgraph Engine Layer
        EE[EmbeddedEngine]
        TX[Transaction Support]
    end
    
    subgraph Storage Layer
        DBM[Dbm Interface]
    end
    
    AM --> MI
    MI --> MR
    MR --> MS
    MR --> TX
    MS --> DBM
    TX --> DBM
    EE --> MR
    EE --> MS
```

## Components

### 1. Migration Interface

The `Migration` interface defines the contract that all migrations must implement.

**File:** `src/Migrations/Migration.vala`

```vala
namespace Implexus.Migrations {

/**
 * Interface for database migrations.
 *
 * Migrations are application-defined classes that modify the database
 * schema or data in a controlled, versioned manner.
 *
 * Each migration must implement:
 * - A unique identifier (version)
 * - An up() method to apply the migration
 * - A down() method to reverse the migration (optional for irreversible migrations)
 *
 * Example:
 * {{{
 * public class CreateUsersTable : Object, Migration {
 *     public string version { get { return "2026031301"; } }
 *     public string description { get { return "Create users container"; } }
 *     
 *     public void up(Core.Engine engine) throws MigrationError {
 *         var root = engine.get_root();
 *         try {
 *             root.create_container("users");
 *         } catch (Core.EngineError e) {
 *             throw new MigrationError.EXECUTION_FAILED(
 *                 "Failed to create users container: %s".printf(e.message)
 *             );
 *         }
 *     }
 *     
 *     public void down(Core.Engine engine) throws MigrationError {
 *         var root = engine.get_root();
 *         try {
 *             var users = root.get_child("users");
 *             if (users != null) {
 *                 users.delete();
 *             }
 *         } catch (Core.EngineError e) {
 *             throw new MigrationError.EXECUTION_FAILED(
 *                 "Failed to delete users container: %s".printf(e.message)
 *             );
 *         }
 *     }
 * }
 * }}}
 */
public interface Migration : Object {
    
    /**
     * Unique identifier for this migration.
     *
     * Best practice: Use a timestamp format like YYYYMMDDNN where NN is a
     * sequence number within the day. This ensures proper ordering.
     *
     * Examples: "2026031301", "2026031302", "2026031501"
     */
    public abstract string version { owned get; }
    
    /**
     * Human-readable description of what this migration does.
     */
    public abstract string description { owned get; }
    
    /**
     * Applies the migration.
     *
     * This method is called when migrating forward. All operations
     * are executed within a single transaction.
     *
     * @param engine The database engine to operate on
     * @throws MigrationError if the migration fails
     */
    public abstract void up(Core.Engine engine) throws MigrationError;
    
    /**
     * Reverses the migration.
     *
     * This method is called when rolling back. All operations
     * are executed within a single transaction.
     *
     * Implementations may throw MigrationError.IRREVERSIBLE if the
     * migration cannot be safely reversed.
     *
     * @param engine The database engine to operate on
     * @throws MigrationError if the rollback fails
     */
    public abstract void down(Core.Engine engine) throws MigrationError;
}

} // namespace Implexus.Migrations
```

### 2. MigrationRunner

The `MigrationRunner` discovers and executes migrations in the correct order.

**File:** `src/Migrations/MigrationRunner.vala`

```vala
namespace Implexus.Migrations {

/**
 * Delegate for migration progress notifications.
 */
public delegate void MigrationProgressDelegate(Migration migration, bool is_up);

/**
 * Discovers and executes migrations in version order.
 *
 * The MigrationRunner is responsible for:
 * - Registering available migrations
 * - Determining which migrations need to run
 * - Executing migrations in the correct order within transactions
 * - Recording migration history
 *
 * Example usage:
 * {{{
 * var runner = new MigrationRunner(engine);
 * 
 * // Register migrations
 * runner.register_migration(new CreateUsersTable());
 * runner.register_migration(new AddEmailIndex());
 * runner.register_migration(new SeedInitialData());
 * 
 * // Run all pending migrations
 * try {
 *     int count = runner.run_pending();
 *     print("Ran %d migrations\n", count);
 * } catch (MigrationError e) {
 *     stderr.printf("Migration failed: %s\n", e.message);
 * }
 * }}}
 */
public class MigrationRunner : Object {
    
    // === Private Fields ===
    
    private weak Core.Engine _engine;
    private MigrationStorage _storage;
    private Invercargill.DataStructures.Dictionary<string, Migration> _migrations;
    
    // === Constructors ===
    
    /**
     * Creates a new MigrationRunner for the given engine.
     *
     * @param engine The database engine to migrate
     */
    public MigrationRunner(Core.Engine engine) {
        _engine = engine;
        _storage = new MigrationStorage(engine);
        _migrations = new Invercargill.DataStructures.Dictionary<string, Migration>();
    }
    
    // === Migration Registration ===
    
    /**
     * Registers a migration to be available for execution.
     *
     * Migrations must be registered before calling run_pending() or rollback_to().
     *
     * @param migration The migration to register
     * @throws MigrationError.VERSION_CONFLICT if a migration with the same version exists
     */
    public void register_migration(Migration migration) throws MigrationError {
        var version = migration.version;
        if (_migrations.has(version)) {
            throw new MigrationError.VERSION_CONFLICT(
                "Migration version %s is already registered".printf(version)
            );
        }
        _migrations.set(version, migration);
    }
    
    /**
     * Registers multiple migrations at once.
     *
     * @param migrations Array of migrations to register
     * @throws MigrationError if any registration fails
     */
    public void register_migrations(Migration[] migrations) throws MigrationError {
        foreach (var migration in migrations) {
            register_migration(migration);
        }
    }
    
    // === Status Queries ===
    
    /**
     * Gets all migration versions that have been applied.
     *
     * @return Sorted list of applied migration versions
     */
    public Invercargill.Enumerable<string> get_applied_versions() {
        return _storage.get_applied_versions();
    }
    
    /**
     * Gets all migration versions that are registered but not yet applied.
     *
     * @return Sorted list of pending migration versions
     */
    public Invercargill.Enumerable<string> get_pending_versions() {
        var pending = new Invercargill.DataStructures.Vector<string>();
        var applied = _storage.get_applied_set();
        
        // Get all registered versions and sort them
        var all_versions = new Invercargill.DataStructures.Vector<string>();
        foreach (var version in _migrations.keys) {
            all_versions.add(version);
        }
        all_versions.sort((a, b) => a.compare(b));
        
        // Filter to only pending
        foreach (var version in all_versions) {
            if (!applied.contains(version)) {
                pending.add(version);
            }
        }
        
        return pending.as_enumerable();
    }
    
    /**
     * Checks if a specific migration version has been applied.
     *
     * @param version The migration version to check
     * @return true if the migration has been applied
     */
    public bool is_applied(string version) {
        return _storage.is_applied(version);
    }
    
    /**
     * Gets the count of pending migrations.
     *
     * @return Number of migrations waiting to be applied
     */
    public int get_pending_count() {
        int count = 0;
        var applied = _storage.get_applied_set();
        foreach (var version in _migrations.keys) {
            if (!applied.contains(version)) {
                count++;
            }
        }
        return count;
    }
    
    // === Execution ===
    
    /**
     * Runs all pending migrations in version order.
     *
     * Each migration runs in its own transaction. If a migration fails,
     * the process stops and the failing migration is rolled back.
     *
     * @param progress Optional callback for progress notifications
     * @return Number of migrations that were run
     * @throws MigrationError if any migration fails
     */
    public int run_pending(MigrationProgressDelegate? progress = null) throws MigrationError {
        var pending = get_pending_versions();
        int count = 0;
        
        foreach (var version in pending) {
            var migration = _migrations.get(version);
            if (migration == null) {
                throw new MigrationError.NOT_FOUND(
                    "Migration %s not found".printf(version)
                );
            }
            
            run_single((!) migration, true, progress);
            count++;
        }
        
        return count;
    }
    
    /**
     * Runs a specific migration.
     *
     * @param version The version of the migration to run
     * @param progress Optional callback for progress notification
     * @throws MigrationError if the migration fails or is already applied
     */
    public void run_one(string version, MigrationProgressDelegate? progress = null) throws MigrationError {
        if (_storage.is_applied(version)) {
            throw new MigrationError.ALREADY_APPLIED(
                "Migration %s is already applied".printf(version)
            );
        }
        
        var migration = _migrations.get(version);
        if (migration == null) {
            throw new MigrationError.NOT_FOUND(
                "Migration %s not found".printf(version)
            );
        }
        
        run_single((!) migration, true, progress);
    }
    
    /**
     * Rolls back to a specific version.
     *
     * Runs the down() method of all migrations after the target version,
     * in reverse order.
     *
     * @param target_version The version to roll back to (exclusive - this version remains applied)
     * @param progress Optional callback for progress notifications
     * @return Number of migrations that were rolled back
     * @throws MigrationError if any rollback fails
     */
    public int rollback_to(string target_version, MigrationProgressDelegate? progress = null) throws MigrationError {
        var applied = _storage.get_applied_versions();
        int count = 0;
        
        // Roll back in reverse order
        var to_rollback = new Invercargill.DataStructures.Vector<string>();
        foreach (var version in applied) {
            if (version.compare(target_version) > 0) {
                to_rollback.add(version);
            }
        }
        
        // Sort in reverse order
        to_rollback.sort((a, b) => b.compare(a));
        
        foreach (var version in to_rollback) {
            var migration = _migrations.get(version);
            if (migration == null) {
                throw new MigrationError.NOT_FOUND(
                    "Migration %s not found for rollback".printf(version)
                );
            }
            
            run_single((!) migration, false, progress);
            count++;
        }
        
        return count;
    }
    
    /**
     * Rolls back the most recently applied migration.
     *
     * @param progress Optional callback for progress notification
     * @throws MigrationError if no migrations are applied or rollback fails
     */
    public void rollback_last(MigrationProgressDelegate? progress = null) throws MigrationError {
        var applied = _storage.get_applied_versions();
        
        string? last_version = null;
        foreach (var version in applied) {
            last_version = version; // Last one due to sorted order
        }
        
        if (last_version == null) {
            throw new MigrationError.NO_MIGRATIONS(
                "No migrations have been applied"
            );
        }
        
        var migration = _migrations.get((!) last_version);
        if (migration == null) {
            throw new MigrationError.NOT_FOUND(
                "Migration %s not found for rollback".printf((!) last_version)
            );
        }
        
        run_single((!) migration, false, progress);
    }
    
    // === Internal Methods ===
    
    /**
     * Runs a single migration within a transaction.
     */
    private void run_single(Migration migration, bool is_up, MigrationProgressDelegate? progress) 
            throws MigrationError {
        
        var engine = _engine;
        if (engine == null) {
            throw new MigrationError.ENGINE_ERROR("Engine reference is invalid");
        }
        
        // Execute within transaction
        try {
            var tx = ((!) engine).begin_transaction();
            try {
                if (is_up) {
                    migration.up((!) engine);
                    _storage.record_migration(migration.version, migration.description);
                } else {
                    migration.down((!) engine);
                    _storage.remove_migration(migration.version);
                }
                
                tx.commit();
                
                if (progress != null) {
                    progress(migration, is_up);
                }
            } catch (MigrationError e) {
                tx.rollback();
                throw e;
            } catch (Core.EngineError e) {
                tx.rollback();
                throw new MigrationError.EXECUTION_FAILED(
                    "Migration %s failed: %s".printf(migration.version, e.message)
                );
            }
        } catch (Core.EngineError e) {
            throw new MigrationError.TRANSACTION_ERROR(
                "Failed to begin transaction: %s".printf(e.message)
            );
        }
    }
}

} // namespace Implexus.Migrations
```

### 3. MigrationStorage

The `MigrationStorage` class handles persistence of migration history using the existing Dbm infrastructure.

**File:** `src/Migrations/MigrationStorage.vala`

```vala
namespace Implexus.Migrations {

/**
 * Stores and retrieves migration history.
 *
 * Migration history is persisted in the database using a dedicated
 * key prefix. This ensures migration state survives application restarts.
 *
 * Key format: migration:<version>
 * Value: Serialized (timestamp, description)
 */
public class MigrationStorage : Object {
    
    // === Constants ===
    
    private const string PREFIX = "migration:";
    private const string INDEX_KEY = "migration:index";
    
    // === Private Fields ===
    
    private weak Core.Engine _engine;
    private Storage.Dbm? _dbm;
    
    // === Constructors ===
    
    /**
     * Creates a new MigrationStorage for the given engine.
     *
     * @param engine The database engine
     */
    public MigrationStorage(Core.Engine engine) {
        _engine = engine;
        
        // Get Dbm from engine configuration if available
        var config = engine.configuration;
        var storage = config.storage;
        
        // Try to get Dbm from BasicStorage
        var basic_storage = (storage as Storage.BasicStorage);
        if (basic_storage != null) {
            _dbm = ((!) basic_storage).dbm;
        }
    }
    
    // === Recording Migrations ===
    
    /**
     * Records that a migration has been applied.
     *
     * @param version The migration version
     * @param description The migration description
     * @throws MigrationError if the record cannot be saved
     */
    public void record_migration(string version, string description) throws MigrationError {
        if (_dbm == null) {
            throw new MigrationError.STORAGE_ERROR("Database storage not available");
        }
        
        var dbm = (!) _dbm;
        string key = PREFIX + version;
        
        // Store migration record
        var writer = new Storage.ElementWriter();
        writer.write_element(new Invercargill.NativeElement<int64?>(new DateTime.now_utc().to_unix()));
        writer.write_element(new Invercargill.NativeElement<string>(description));
        
        try {
            dbm.set(key, writer.to_binary_data());
        } catch (Storage.StorageError e) {
            throw new MigrationError.STORAGE_ERROR(
                "Failed to record migration: %s".printf(e.message)
            );
        }
        
        // Update index
        update_index(dbm, version, true);
    }
    
    /**
     * Removes a migration record (for rollbacks).
     *
     * @param version The migration version to remove
     * @throws MigrationError if the record cannot be removed
     */
    public void remove_migration(string version) throws MigrationError {
        if (_dbm == null) {
            throw new MigrationError.STORAGE_ERROR("Database storage not available");
        }
        
        var dbm = (!) _dbm;
        string key = PREFIX + version;
        
        try {
            dbm.delete(key);
        } catch (Storage.StorageError e) {
            // Key may not exist, that's fine
        }
        
        // Update index
        update_index(dbm, version, false);
    }
    
    // === Querying Migrations ===
    
    /**
     * Gets all applied migration versions in sorted order.
     *
     * @return Sorted enumerable of version strings
     */
    public Invercargill.Enumerable<string> get_applied_versions() {
        var versions = new Invercargill.DataStructures.Vector<string>();
        
        if (_dbm == null) {
            return versions.as_enumerable();
        }
        
        // Load from index
        var index_data = ((!) _dbm).get(INDEX_KEY);
        if (index_data == null) {
            // Fall back to scanning keys
            foreach (var key in ((!) _dbm).keys) {
                if (key.has_prefix(PREFIX)) {
                    var version = key.substring(PREFIX.length);
                    versions.add(version);
                }
            }
            versions.sort((a, b) => a.compare(b));
            return versions.as_enumerable();
        }
        
        // Parse index
        var reader = new Storage.ElementReader((!) index_data);
        try {
            var element = reader.read_element();
            if (!element.is_null()) {
                var array = element.as<Invercargill.Enumerable<Invercargill.Element>>();
                foreach (var item in array) {
                    if (!item.is_null()) {
                        versions.add(item.as<string>());
                    }
                }
            }
        } catch (Invercargill.ElementError e) {
            // Fall back to empty list
        }
        
        return versions.as_enumerable();
    }
    
    /**
     * Gets a set of applied migration versions for efficient lookup.
     *
     * @return Set of applied version strings
     */
    public Invercargill.DataStructures.Set<string> get_applied_set() {
        var set = new Invercargill.DataStructures.HashSet<string>();
        
        foreach (var version in get_applied_versions()) {
            set.add(version);
        }
        
        return set;
    }
    
    /**
     * Checks if a specific migration has been applied.
     *
     * @param version The migration version to check
     * @return true if the migration has been applied
     */
    public bool is_applied(string version) {
        if (_dbm == null) {
            return false;
        }
        
        string key = PREFIX + version;
        return ((!) _dbm).has_key(key);
    }
    
    /**
     * Gets detailed information about an applied migration.
     *
     * @param version The migration version
     * @return Migration record, or null if not applied
     */
    public MigrationRecord? get_migration_record(string version) {
        if (_dbm == null) {
            return null;
        }
        
        string key = PREFIX + version;
        var data = ((!) _dbm).get(key);
        
        if (data == null) {
            return null;
        }
        
        var reader = new Storage.ElementReader((!) data);
        try {
            var timestamp_element = reader.read_element();
            var desc_element = reader.read_element();
            
            int64? timestamp = timestamp_element.as<int64?>();
            string description = desc_element.as<string>();
            
            return new MigrationRecord(
                version,
                description,
                timestamp != null ? new DateTime.from_unix_utc((!) timestamp) : null
            );
        } catch (Invercargill.ElementError e) {
            return null;
        }
    }
    
    // === Private Methods ===
    
    /**
     * Updates the migration index.
     */
    private void update_index(Storage.Dbm dbm, string version, bool add) {
        var versions = new Invercargill.DataStructures.Vector<string>();
        
        // Load existing index
        var index_data = dbm.get(INDEX_KEY);
        if (index_data != null) {
            var reader = new Storage.ElementReader((!) index_data);
            try {
                var element = reader.read_element();
                if (!element.is_null()) {
                    var array = element.as<Invercargill.Enumerable<Invercargill.Element>>();
                    foreach (var item in array) {
                        if (!item.is_null()) {
                            string existing = item.as<string>();
                            if (!add || existing != version) {
                                versions.add(existing);
                            }
                        }
                    }
                }
            } catch (Invercargill.ElementError e) {
                // Start fresh
            }
        }
        
        if (add) {
            versions.add(version);
        }
        
        // Sort
        versions.sort((a, b) => a.compare(b));
        
        // Save index
        if (versions.length == 0) {
            try {
                dbm.delete(INDEX_KEY);
            } catch (Storage.StorageError e) {
                // Ignore
            }
            return;
        }
        
        var array = new Invercargill.DataStructures.Vector<Invercargill.Element>();
        foreach (var v in versions) {
            array.add(new Invercargill.NativeElement<string>(v));
        }
        
        var writer = new Storage.ElementWriter();
        writer.write_element(new Invercargill.NativeElement<Invercargill.Enumerable<Invercargill.Element>>(array));
        
        try {
            dbm.set(INDEX_KEY, writer.to_binary_data());
        } catch (Storage.StorageError e) {
            // Ignore - not critical
        }
    }
}

/**
 * Represents a record of an applied migration.
 */
public class MigrationRecord : Object {
    /**
     * The migration version.
     */
    public string version { get; construct set; }
    
    /**
     * The migration description.
     */
    public string description { get; construct set; }
    
    /**
     * When the migration was applied, or null if unknown.
     */
    public DateTime? applied_at { get; construct set; }
    
    /**
     * Creates a new MigrationRecord.
     */
    public MigrationRecord(string version, string description, DateTime? applied_at) {
        Object(version: version, description: description, applied_at: applied_at);
    }
}

} // namespace Implexus.Migrations
```

### 4. MigrationError

Error domain for migration-related errors.

**File:** `src/Migrations/MigrationError.vala`

```vala
namespace Implexus.Migrations {

/**
 * Error domain for migration operations.
 */
public errordomain MigrationError {
    /**
     * The migration was not found.
     */
    NOT_FOUND,
    
    /**
     * A migration with this version is already registered or applied.
     */
    VERSION_CONFLICT,
    
    /**
     * The migration is already applied.
     */
    ALREADY_APPLIED,
    
    /**
     * The migration cannot be reversed.
     */
    IRREVERSIBLE,
    
    /**
     * The migration execution failed.
     */
    EXECUTION_FAILED,
    
    /**
     * Transaction error during migration.
     */
    TRANSACTION_ERROR,
    
    /**
     * Storage error during migration.
     */
    STORAGE_ERROR,
    
    /**
     * Engine reference error.
     */
    ENGINE_ERROR,
    
    /**
     * No migrations are available or applied.
     */
    NO_MIGRATIONS
}

} // namespace Implexus.Migrations
```

### 5. Bootstrap Migration

A special migration for initial database setup.

**File:** `src/Migrations/BootstrapMigration.vala`

```vala
namespace Implexus.Migrations {

/**
 * A bootstrap migration that runs before any other migrations.
 *
 * This migration is automatically applied when the database is first
 * created. It can be extended by applications to perform initial setup
 * such as creating required containers or seeding default data.
 *
 * The bootstrap migration always uses version "0000000000" to ensure
 * it runs first.
 *
 * Example:
 * {{{
 * public class MyAppBootstrap : BootstrapMigration {
 *     public override void up(Core.Engine engine) throws MigrationError {
 *         base.up(engine);
 *         
 *         var root = engine.get_root();
 *         try {
 *             // Create application-specific containers
 *             root.create_container("config");
 *             root.create_container("sessions");
 *             
 *             // Seed initial data
 *             var config = root.get_child("config");
 *             var settings = config.create_document("settings", "AppConfig");
 *             settings.set_entity_property("version", new Invercargill.NativeElement<string>("1.0.0"));
 *         } catch (Core.EngineError e) {
 *             throw new MigrationError.EXECUTION_FAILED(
 *                 "Bootstrap failed: %s".printf(e.message)
 *             );
 *         }
 *     }
 * }
 * }}}
 */
public class BootstrapMigration : Object, Migration {
    
    /**
     * Bootstrap version - always runs first.
     */
    public string version { owned get { return "0000000000"; } }
    
    /**
     * Description of the bootstrap migration.
     */
    public virtual string description { owned get { return "Initial database setup"; } }
    
    /**
     * Performs initial database setup.
     *
     * The base implementation ensures the root container exists.
     * Override to add application-specific setup.
     */
    public virtual void up(Core.Engine engine) throws MigrationError {
        // Ensure root exists
        try {
            engine.get_root();
        } catch (Core.EngineError e) {
            throw new MigrationError.EXECUTION_FAILED(
                "Failed to initialize root: %s".printf(e.message)
            );
        }
    }
    
    /**
     * Bootstrap migration cannot be reversed.
     */
    public void down(Core.Engine engine) throws MigrationError {
        throw new MigrationError.IRREVERSIBLE(
            "Bootstrap migration cannot be reversed"
        );
    }
}

} // namespace Implexus.Migrations
```

## File Organization

New files should be placed in the following structure:

```
src/
├── Migration/
│   ├── meson.build
│   ├── Migration.vala              # Migration interface
│   ├── MigrationRunner.vala        # Migration execution engine
│   ├── MigrationStorage.vala       # History persistence
│   ├── MigrationError.vala         # Error domain
│   └── BootstrapMigration.vala     # Initial setup migration
```

### meson.build

```meson
migration_sources = files(
    'Migration.vala',
    'MigrationRunner.vala',
    'MigrationStorage.vala',
    'MigrationError.vala',
    'BootstrapMigration.vala'
)
```

The migration sources should be added to the main library sources in `src/meson.build`.

## Integration with Engine

### Engine Interface Extension

Add migration support to the Engine interface:

```vala
// In src/Core/Engine.vala

public interface Engine : Object {
    // ... existing methods ...
    
    /**
     * Creates a new MigrationRunner for this engine.
     *
     * @return A new MigrationRunner instance
     */
    public abstract Migration.MigrationRunner create_migration_runner();
}
```

### EmbeddedEngine Implementation

```vala
// In src/Engine/EmbeddedEngine.vala

public class EmbeddedEngine : Object, Core.Engine {
    // ... existing code ...
    
    /**
     * {@inheritDoc}
     */
    public Migration.MigrationRunner create_migration_runner() {
        return new Migration.MigrationRunner(this);
    }
}
```

## API Examples

### Defining Migrations

```vala
// Migration 1: Create users container
public class CreateUsersContainer : Object, Implexus.Migrations.Migration {
    public string version { owned get { return "2026031301"; } }
    public string description { owned get { return "Create users container"; } }
    
    public void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        try {
            var root = engine.get_root();
            root.create_container("users");
        } catch (Implexus.Core.EngineError e) {
            throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
                "Failed to create users: %s".printf(e.message)
            );
        }
    }
    
    public void down(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        try {
            var root = engine.get_root();
            var users = root.get_child("users");
            if (users != null) {
                users.delete();
            }
        } catch (Implexus.Core.EngineError e) {
            throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
                "Failed to remove users: %s".printf(e.message)
            );
        }
    }
}

// Migration 2: Add email index
public class AddEmailIndex : Object, Implexus.Migrations.Migration {
    public string version { owned get { return "2026031302"; } }
    public string description { owned get { return "Add email index for users"; } }
    
    public void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        try {
            var root = engine.get_root();
            var users = (Implexus.Entities.Container?) root.get_child("users");
            if (users != null) {
                ((!) users).create_index("by_email", "User", "email");
            }
        } catch (Implexus.Core.EngineError e) {
            throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
                "Failed to create email index: %s".printf(e.message)
            );
        }
    }
    
    public void down(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        try {
            var root = engine.get_root();
            var users = root.get_child("users");
            if (users != null) {
                var index = ((!) users).get_child("by_email");
                if (index != null) {
                    index.delete();
                }
            }
        } catch (Implexus.Core.EngineError e) {
            throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
                "Failed to remove email index: %s".printf(e.message)
            );
        }
    }
}

// Migration 3: Seed initial data (irreversible)
public class SeedInitialData : Object, Implexus.Migrations.Migration {
    public string version { owned get { return "2026031303"; } }
    public string description { owned get { return "Seed initial configuration data"; } }
    
    public void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        try {
            var root = engine.get_root();
            var config = root.create_container("config");
            var settings = config.create_document("settings", "AppSettings");
            settings.set_entity_property("theme", new Invercargill.NativeElement<string>("dark"));
            settings.set_entity_property("max_users", new Invercargill.NativeElement<int>(100));
        } catch (Implexus.Core.EngineError e) {
            throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
                "Failed to seed data: %s".printf(e.message)
            );
        }
    }
    
    public void down(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        // This migration is irreversible - data loss is not acceptable
        throw new Implexus.Migrations.MigrationError.IRREVERSIBLE(
            "Seed data migration cannot be reversed"
        );
    }
}
```

### Running Migrations

```vala
// Basic migration run
public void run_migrations(Implexus.Core.Engine engine) {
    var runner = engine.create_migration_runner();
    
    try {
        // Register all migrations
        runner.register_migration(new CreateUsersContainer());
        runner.register_migration(new AddEmailIndex());
        runner.register_migration(new SeedInitialData());
        
        // Run pending migrations
        int count = runner.run_pending();
        print("Successfully ran %d migrations\n", count);
        
    } catch (Implexus.Migrations.MigrationError e) {
        stderr.printf("Migration failed: %s\n", e.message);
    }
}

// With progress reporting
public void run_migrations_with_progress(Implexus.Core.Engine engine) {
    var runner = engine.create_migration_runner();
    
    try {
        runner.register_migrations(new Implexus.Migrations.Migration[] {
            new CreateUsersContainer(),
            new AddEmailIndex(),
            new SeedInitialData()
        });
        
        // Run with progress callback
        int count = runner.run_pending((migration, is_up) => {
            string direction = is_up ? "Up" : "Down";
            print("[%s] %s: %s\n", direction, migration.version, migration.description);
        });
        
        print("Completed %d migrations\n", count);
        
    } catch (Implexus.Migrations.MigrationError e) {
        stderr.printf("Migration failed: %s\n", e.message);
    }
}

// Check status before running
public void check_and_run_migrations(Implexus.Core.Engine engine) {
    var runner = engine.create_migration_runner();
    
    try {
        runner.register_migrations(new Implexus.Migrations.Migration[] {
            new CreateUsersContainer(),
            new AddEmailIndex(),
            new SeedInitialData()
        });
        
        // Check status
        int pending = runner.get_pending_count();
        if (pending == 0) {
            print("Database is up to date\n");
            return;
        }
        
        print("There are %d pending migrations:\n", pending);
        foreach (var version in runner.get_pending_versions()) {
            print("  - %s\n", version);
        }
        
        // Run them
        runner.run_pending();
        
    } catch (Implexus.Migrations.MigrationError e) {
        stderr.printf("Migration failed: %s\n", e.message);
    }
}
```

### Rolling Back Migrations

```vala
// Roll back last migration
public void rollback_last_migration(Implexus.Core.Engine engine) {
    var runner = engine.create_migration_runner();
    
    try {
        runner.register_migrations(new Implexus.Migrations.Migration[] {
            new CreateUsersContainer(),
            new AddEmailIndex(),
            new SeedInitialData()
        });
        
        runner.rollback_last((migration, is_up) => {
            print("Rolled back: %s\n", migration.version);
        });
        
    } catch (Implexus.Migrations.MigrationError e) {
        if (e is Implexus.Migrations.MigrationError.IRREVERSIBLE) {
            stderr.printf("Cannot roll back irreversible migration: %s\n", e.message);
        } else {
            stderr.printf("Rollback failed: %s\n", e.message);
        }
    }
}

// Roll back to specific version
public void rollback_to_version(Implexus.Core.Engine engine, string target_version) {
    var runner = engine.create_migration_runner();
    
    try {
        runner.register_migrations(new Implexus.Migrations.Migration[] {
            new CreateUsersContainer(),
            new AddEmailIndex(),
            new SeedInitialData()
        });
        
        int count = runner.rollback_to(target_version);
        print("Rolled back %d migrations\n", count);
        
    } catch (Implexus.Migrations.MigrationError e) {
        stderr.printf("Rollback failed: %s\n", e.message);
    }
}
```

### Using Bootstrap Migration

```vala
// Custom bootstrap for application
public class MyAppBootstrap : Implexus.Migrations.BootstrapMigration {
    public override string description { owned get { return "MyApp initial setup"; } }
    
    public override void up(Implexus.Core.Engine engine) throws Implexus.Migrations.MigrationError {
        // Call base to ensure root exists
        base.up(engine);
        
        try {
            var root = engine.get_root();
            
            // Create application structure
            root.create_container("users");
            root.create_container("projects");
            root.create_container("settings");
            
            // Create default admin user
            var users = (Implexus.Entities.Container) root.get_child("users");
            var admin = users.create_document("admin", "User");
            admin.set_entity_property("role", new Invercargill.NativeElement<string>("admin"));
            admin.set_entity_property("email", new Invercargill.NativeElement<string>("admin@example.com"));
            
        } catch (Implexus.Core.EngineError e) {
            throw new Implexus.Migrations.MigrationError.EXECUTION_FAILED(
                "Bootstrap failed: %s".printf(e.message)
            );
        }
    }
}

// Using bootstrap with regular migrations
public void initialize_database(Implexus.Core.Engine engine) {
    var runner = engine.create_migration_runner();
    
    try {
        // Register bootstrap first
        runner.register_migration(new MyAppBootstrap());
        
        // Then register regular migrations
        runner.register_migration(new CreateUsersContainer());
        runner.register_migration(new AddEmailIndex());
        
        // Run all pending (bootstrap will run first due to version ordering)
        runner.run_pending();
        
    } catch (Implexus.Migrations.MigrationError e) {
        stderr.printf("Database initialization failed: %s\n", e.message);
    }
}
```

## Error Handling

### Error Handling Strategy

1. **Transaction Safety**: Each migration runs in its own transaction. If a migration fails, only that migration is rolled back.

2. **Atomic Operations**: Within a migration, all operations are atomic. Either all changes apply, or none do.

3. **Irreversible Migrations**: Migrations that cannot be safely reversed should throw `MigrationError.IRREVERSIBLE` in their `down()` method.

4. **Progress Preservation**: Successfully applied migrations are recorded even if later migrations fail. The system can be resumed after fixing the failing migration.

### Error Handling Flow

```mermaid
sequenceDiagram
    participant App as Application
    participant Runner as MigrationRunner
    participant TX as Transaction
    participant Storage as MigrationStorage
    participant DB as Database
    
    App->>Runner: run_pending()
    loop For each pending migration
        Runner->>TX: begin_transaction()
        Runner->>App: migration.up()
        alt Success
            App->>DB: Apply changes
            Runner->>Storage: record_migration()
            Storage->>DB: Save record
            Runner->>TX: commit()
            Runner->>App: progress callback
        else Failure
            App-->>Runner: throws EngineError
            Runner->>TX: rollback()
            Runner-->>App: throws MigrationError
        end
    end
```

### Error Recovery

```vala
public void run_with_recovery(Implexus.Core.Engine engine) {
    var runner = engine.create_migration_runner();
    
    try {
        runner.register_migrations(get_all_migrations());
        runner.run_pending();
        
    } catch (Implexus.Migrations.MigrationError e) {
        // Check what was applied before failure
        var applied = runner.get_applied_versions();
        var pending = runner.get_pending_versions();
        
        stderr.printf("Migration failed: %s\n", e.message);
        stderr.printf("Applied: %d, Pending: %d\n", 
            applied.length, pending.length);
        
        // Log for manual recovery
        log_migration_failure(e, applied, pending);
        
        // Depending on error type, may be able to retry
        if (e is Implexus.Migrations.MigrationError.TRANSACTION_ERROR) {
            stderr.printf("Transaction error - may be retried\n");
        } else if (e is Implexus.Migrations.MigrationError.EXECUTION_FAILED) {
            stderr.printf("Execution failed - check migration code\n");
        }
    }
}
```

## Storage Schema

### Key Schema

| Key Pattern | Description |
|-------------|-------------|
| `migration:<version>` | Record of an applied migration |
| `migration:index` | Sorted list of all applied migration versions |

### Data Format

**migration:<version>**
```
Element[] {
    int64 timestamp,      // Unix timestamp when applied
    string description    // Migration description
}
```

**migration:index**
```
Element[] {
    string[] versions    // Sorted array of applied version strings
}
```

## Design Decisions

1. **Version Format**: Using string-based versions (e.g., "2026031301") instead of integers allows for:
   - Natural ordering by timestamp
   - Human-readable identification
   - No collision issues in distributed development

2. **One Transaction Per Migration**: Each migration runs in its own transaction rather than batching all migrations in one transaction. This:
   - Allows partial progress recovery
   - Prevents long-running transactions
   - Matches common migration tool patterns

3. **Separate MigrationStorage**: Storage logic is separated from the runner to:
   - Allow alternative storage implementations
   - Enable testing with mock storage
   - Follow single responsibility principle

4. **Bootstrap as Regular Migration**: The bootstrap migration is implemented as a regular migration with a special version number rather than a separate mechanism. This:
   - Ensures it's tracked in history
   - Allows rollback (if reversible)
   - Uses the same infrastructure

5. **No Auto-Discovery**: Migrations must be explicitly registered rather than auto-discovered. This:
   - Gives applications full control
   - Avoids reflection/metadata complexity
   - Works well with Vala's compilation model

## Future Considerations

1. **Migration Validation**: Could add a `validate()` method to migrations for pre-flight checks.

2. **Dry Run Mode**: Could add a `--dry-run` option that shows what would happen without executing.

3. **Migration Dependencies**: Could add support for migrations that depend on other migrations.

4. **Conditional Migrations**: Could add support for migrations that only run under certain conditions.

5. **Migration Groups**: Could add support for grouping migrations into named sets that can be applied together.