Storage Layer Migration Plan

Overview

Migrate from legacy BasicStorage + IndexManager to the new HighLevel + LowLevel architecture while preserving all performance optimizations.

Current State Analysis

Legacy Components (to be removed)

BasicStorage - High-level storage interface
IndexManager - Index operations with performance optimizations

New Architecture (to be completed)

LowLevel: Single-responsibility storage classes per key prefix
HighLevel: Entity-specific facades composing LowLevel stores

Performance Optimizations in IndexManager (must preserve)

Optimization	Location	Description
HashSet dedup on load	`load_string_set()` L764-770	Uses HashSet during deserialization to deduplicate while preserving order
HashSet for membership checks	`add_to_ngram_index()` L407-408	Creates HashSet for O(1) contains() checks instead of O(n) Vector.contains()
HashSet for remove	`remove_from_ngram_index()` L428-429	HashSet for efficient membership test before rebuild
Batch add with HashSet	`add_to_ngram_index_batch()` L446-456	Tracks changes, only saves if modified, uses HashSet for dedup
Batch remove with HashSet	`remove_from_ngram_index_batch()` L463-474	HashSet for values, rebuilds vector without matches
Set members with dedup	`set_category_members()` L205-213	Uses HashSet to deduplicate input enumerable
Batch reverse index	`add_bigrams_reverse_batch()` L564-570	Dictionary-based batch operations
Trigram batch ops	`add_trigrams_batch()` L622-627	Dictionary-based batch trigram operations

Migration Plan

Phase 1: Add Batch Operations to LowLevel Classes

Add the missing batch methods and HashSet optimizations to LowLevel storage classes.

1.1 Update `CategoryIndexStorage`

File: src/Storage/LowLevel/CategoryIndexStorage.vala

Changes:

Add set_members() method with HashSet deduplication (from IndexManager L205-213)
Optimize add_member() to use HashSet for O(1) membership check
Optimize remove_member() to use HashSet for O(1) membership check

Current vs Optimized:

// Current (O(n) contains check)
public void add_member(string category_path, string doc_path) throws StorageError {
    string key = members_key(category_path);
    var members = load_string_set(key);
    if (!members.contains(doc_path)) {  // O(n) operation
        members.add(doc_path);
        save_string_set(key, members);
    }
}

// Optimized (O(1) contains check)
public void add_member(string category_path, string doc_path) throws StorageError {
    string key = members_key(category_path);
    var members = load_string_set(key);
    var members_hash = new Invercargill.DataStructures.HashSet<string>();
    foreach (var m in members) members_hash.add(m);
    
    if (!members_hash.has(doc_path)) {  // O(1) operation
        members.add(doc_path);
        save_string_set(key, members);
    }
}

1.2 Update `CatalogueIndexStorage`

File: src/Storage/LowLevel/CatalogueIndexStorage.vala

Changes:

Add set_group_members() method with HashSet deduplication
Optimize add_to_group() with HashSet membership check
Optimize remove_from_group() with HashSet membership check

1.3 Update `TextIndexStorage` (Critical - most complex)

File: src/Storage/LowLevel/TextIndexStorage.vala

Changes:

Optimize add_trigram() with HashSet membership check (from IndexManager L404-414)
Optimize remove_trigram() with HashSet membership check (from IndexManager L425-440)
Add add_trigram_batch() method (from IndexManager L442-457)
Add remove_trigram_batch() method (from IndexManager L459-475)
Add add_bigram_mapping_batch() method (from IndexManager L564-570)
Add add_unigram_mapping_batch() method (from IndexManager L614-620)
Add add_trigrams_batch() dictionary method (from IndexManager L622-628)
Add remove_trigrams_batch() dictionary method (from IndexManager L630-636)
Update load_string_set() to use HashSet for deduplication (from IndexManager L747-779)

1.4 Update `TypeIndexStorage`

File: src/Storage/LowLevel/TypeIndexStorage.vala

Changes:

Optimize add_document() with HashSet membership check
Optimize remove_document() with HashSet membership check
Update load_string_set() to use HashSet for deduplication

Phase 2: Add Batch Methods to HighLevel Stores

Expose the new LowLevel batch methods through the HighLevel facades.

2.1 Update `CategoryStore`

File: src/Storage/HighLevel/CategoryStore.vala

Changes:

Add set_members() facade method
Ensure all LowLevel optimizations are properly delegated

2.2 Update `CatalogueStore`

File: src/Storage/HighLevel/CatalogueStore.vala

Changes:

Add set_group_members() facade method
Ensure all LowLevel optimizations are properly delegated

2.3 Update `IndexStore`

File: src/Storage/HighLevel/IndexStore.vala

Changes:

Add add_trigram_batch() facade method
Add remove_trigram_batch() facade method
Add add_bigram_mappings_batch() facade method
Add add_unigram_mappings_batch() facade method
Add add_trigrams_batch() dictionary method
Add remove_trigrams_batch() dictionary method

Phase 3: Migrate Entity Classes

Update Category, Catalogue, and Index entities to use HighLevel stores instead of IndexManager.

3.1 Migrate `Category` Entity

File: src/Entities/Category.vala

Changes:

Replace get_index_manager() calls with get_category_store() calls
Update populate_index() to use CategoryStore.set_members()
Update add_document() to use CategoryStore.add_member()
Update remove_document() to use CategoryStore.remove_member()
Update contains_document() to use CategoryStore.get_members()
Update batch_update_members() to use CategoryStore methods
Update clear_index() to use CategoryStore.clear_index()

Example migration:

// Before (using IndexManager)
var index_manager = get_index_manager();
if (index_manager != null) {
    ((!) index_manager).add_to_category(_path.to_string(), doc_path);
}

// After (using CategoryStore)
var store = get_category_store();
if (store != null) {
    ((!) store).add_member(_path, doc_path);
}

3.2 Migrate `Catalogue` Entity

File: src/Entities/Catalogue.vala

Changes:

Replace get_index_manager() calls with get_catalogue_store() calls
Update all index operations to use CatalogueStore methods
Update batch operations to use CatalogueStore methods

3.3 Migrate `Index` Entity

File: src/Entities/Index.vala

Changes:

Replace get_index_manager() calls with get_index_store() calls
Update trigram operations to use IndexStore methods
Update batch operations to use IndexStore batch methods
Update search methods to use IndexStore for lookups

Phase 4: Update Engine Configuration

4.1 Update `EngineConfiguration`

File: src/Engine/EngineConfiguration.vala

Changes:

Remove index_manager property (L171)
Add store accessors or remove if handled by EmbeddedEngine

4.2 Update `Core.Engine` interface

File: src/Core/Engine.vala

Changes:

Remove index_manager property (L215-216)

4.3 Update `EmbeddedEngine`

File: src/Engine/EmbeddedEngine.vala

Changes:

Remove _index_manager field (L67)
Remove IndexManager initialization (L189)
Remove _configuration.index_manager assignment (L190)
Keep HighLevel store initialization (L211-217)
Expose stores via properties (already done L121-146)

Phase 5: Update Remaining BasicStorage Usage

5.1 Update `EngineFactory`

File: src/Engine/EngineFactory.vala

Changes:

Replace BasicStorage with direct Dbm + HighLevel stores
Or keep BasicStorage for simple cases, use stores for indexed entities

5.2 Update `RemoteEngine`

File: src/Engine/RemoteEngine.vala

Changes:

Remove placeholder BasicStorage if no longer needed
Or update to use new architecture

5.3 Update `Server`

File: src/Server/Server.vala

Changes:

Update storage initialization to use new architecture

Phase 6: Remove Legacy Code

6.1 Remove Files

Delete src/Storage/Storage.vala (BasicStorage class)
Delete src/Storage/IndexManager.vala

6.2 Update meson.build

File: src/meson.build

Changes:

Remove 'Storage/Storage.vala' from storage_sources (L35)
Remove 'Storage/IndexManager.vala' from storage_sources (L36)
Remove "Legacy storage (deprecated, will be removed)" comment (L34)

Phase 7: Testing

7.1 Add Unit Tests for LowLevel Classes

File: tests/Storage/LowLevelStorageTest.vala (new)

Test CategoryIndexStorage with HashSet optimizations
Test CatalogueIndexStorage with HashSet optimizations
Test TextIndexStorage batch operations
Test TypeIndexStorage with HashSet optimizations

7.2 Add Unit Tests for HighLevel Classes

File: tests/Storage/HighLevelStorageTest.vala (new)

Test CategoryStore facade
Test CatalogueStore facade
Test IndexStore batch methods

7.3 Update Existing Tests

Update StorageTest.vala to remove BasicStorage tests
Run full test suite to verify no regressions

7.4 Performance Verification

Run performance benchmarks before migration
Run performance benchmarks after migration
Verify no performance regression

Key Code Patterns to Preserve

Pattern 1: HashSet for O(1) Membership Checks

// When checking if item exists before add/remove
var set_hash = new Invercargill.DataStructures.HashSet<string>();
foreach (var item in set) set_hash.add(item);

if (!set_hash.has(new_item)) {  // O(1) instead of O(n)
    set.add(new_item);
    save_string_set(key, set);
}

Pattern 2: HashSet Deduplication on Load

// When deserializing, deduplicate while preserving order
var result = new Invercargill.DataStructures.Vector<string>();
var hash_set = new Invercargill.DataStructures.HashSet<string>();

foreach (var item in array) {
    if (!item.is_null()) {
        string value = item.as<string>();
        if (!hash_set.has(value)) {  // Prevent duplicates
            hash_set.add(value);
            result.add(value);
        }
    }
}

Pattern 3: Change Tracking for Batch Operations

// Only save if changes were made
bool changed = false;
var existing_hash = new Invercargill.DataStructures.HashSet<string>();
foreach (var ex in existing) existing_hash.add(ex);

foreach (var val in values) {
    if (!existing_hash.has(val)) {
        existing_hash.add(val);
        existing.add(val);
        changed = true;
    }
}
if (changed) save_string_set(key, existing);  // Only save if modified

Pattern 4: Dictionary-based Batch Operations

// Process multiple keys in batch
public void add_trigrams_batch(string index_path, 
    Invercargill.DataStructures.Dictionary<string, Invercargill.DataStructures.Vector<string>> additions) 
    throws StorageError {
    foreach (var trigram in additions.keys) {
        Invercargill.DataStructures.Vector<string> docs;
        additions.try_get(trigram, out docs);
        add_to_ngram_index_batch(index_path, "tri", trigram, docs);
    }
}

Risk Assessment

Risk	Impact	Mitigation
Performance regression	High	Benchmark before/after, preserve HashSet patterns
Data corruption	Critical	Same key prefixes used, no data migration needed
API breakage	Medium	HighLevel stores already exist, just need to use them
Test coverage	Medium	Add tests for LowLevel classes before migration

Estimated Effort

Phase	Complexity
Phase 1: LowLevel optimizations	Medium - careful pattern copying
Phase 2: HighLevel batch methods	Low - simple facades
Phase 3: Entity migration	Medium - many call sites
Phase 4: Engine configuration	Low - few changes
Phase 5: Remaining usage	Low - few files
Phase 6: Remove legacy	Low - delete files
Phase 7: Testing	Medium - comprehensive testing

Execution Order

Phase 1 first - ensures LowLevel classes have all optimizations
Phase 2 second - exposes optimizations through facades
Phase 7.1-7.2 - test new methods before using them
Phase 3 - migrate entities to use new stores
Phase 7.3 - run existing tests
Phase 4-5 - update engine and remaining code
Phase 7.4 - performance verification
Phase 6 - remove legacy code last

storage-migration-plan.md 13 KB Riwayat Mentahan

Storage Layer Migration Plan

Overview

Current State Analysis

Legacy Components (to be removed)

New Architecture (to be completed)

Performance Optimizations in IndexManager (must preserve)

Migration Plan

Phase 1: Add Batch Operations to LowLevel Classes

1.1 Update CategoryIndexStorage

1.2 Update CatalogueIndexStorage

1.3 Update TextIndexStorage (Critical - most complex)

1.4 Update TypeIndexStorage

Phase 2: Add Batch Methods to HighLevel Stores

2.1 Update CategoryStore

2.2 Update CatalogueStore

2.3 Update IndexStore

Phase 3: Migrate Entity Classes

3.1 Migrate Category Entity

3.2 Migrate Catalogue Entity

3.3 Migrate Index Entity

Phase 4: Update Engine Configuration

4.1 Update EngineConfiguration

4.2 Update Core.Engine interface

4.3 Update EmbeddedEngine

Phase 5: Update Remaining BasicStorage Usage

5.1 Update EngineFactory

5.2 Update RemoteEngine

5.3 Update Server

Phase 6: Remove Legacy Code

6.1 Remove Files

6.2 Update meson.build

Phase 7: Testing

7.1 Add Unit Tests for LowLevel Classes

7.2 Add Unit Tests for HighLevel Classes

7.3 Update Existing Tests

7.4 Performance Verification

Key Code Patterns to Preserve

Pattern 1: HashSet for O(1) Membership Checks

Pattern 2: HashSet Deduplication on Load

Pattern 3: Change Tracking for Batch Operations

Pattern 4: Dictionary-based Batch Operations

Risk Assessment

Estimated Effort

Execution Order

storage-migration-plan.md 13 KB

Riwayat Mentahan

1.1 Update `CategoryIndexStorage`

1.2 Update `CatalogueIndexStorage`

1.3 Update `TextIndexStorage` (Critical - most complex)

1.4 Update `TypeIndexStorage`

2.1 Update `CategoryStore`

2.2 Update `CatalogueStore`

2.3 Update `IndexStore`

3.1 Migrate `Category` Entity

3.2 Migrate `Catalogue` Entity

3.3 Migrate `Index` Entity

4.1 Update `EngineConfiguration`

4.2 Update `Core.Engine` interface

4.3 Update `EmbeddedEngine`

5.1 Update `EngineFactory`

5.2 Update `RemoteEngine`

5.3 Update `Server`