# Storage Layer Redesign

## Document Status
- **Created**: 2026-03-13
- **Status**: Draft
- **Author**: Collaborative design session

## Problems with Current Architecture

### 1. Confusing Separation of Concerns
The current design splits storage operations between `Storage` and `IndexManager` without clear boundaries:
- `Storage` handles entity metadata, properties, children, AND configs
- `IndexManager` handles type indices, category members, catalogue groups, AND n-grams
- It's unclear which class should handle what

### 2. Awkward Coupling in EmbeddedEngine
```vala
// EmbeddedEngine must cast to BasicStorage to get Dbm for IndexManager
var basic_storage = (_storage as Storage.BasicStorage);
if (basic_storage != null) {
    _index_manager = new Storage.IndexManager(basic_storage.dbm);
}
```
This is a code smell indicating the abstraction is broken.

### 3. Unclear Value of Storage Interface
The `Storage` interface has only one implementation (`BasicStorage`). It's unclear what alternative implementations would look like or why an interface is needed.

### 4. Naming Confusion
- `Storage.add_child()` / `get_children()` - Structural child names
- `IndexManager.add_to_category()` / `get_category_members()` - Indexed member documents
- Both deal with "children" but mean different things

### 5. Key Prefix Sprawl
Both classes define key prefixes independently, making it hard to see the overall storage schema.

## Proposed Architecture

### Design Principles

1. **One class per prefix** - Each key prefix gets its own focused class
2. **Entity facades compose prefix stores** - High-level APIs for each entity type
3. **Engine holds facade references** - Clean dependency graph
4. **All stores are concrete classes** - No unnecessary interfaces

### Architecture Overview

```mermaid
graph TB
    subgraph Low-Level Prefix Stores
        EMS[EntityMetadataStorage<br/>entity: prefix]
        PS[PropertiesStorage<br/>props: prefix]
        CS[ChildrenStorage<br/>children: prefix]
        CCS[CategoryConfigStorage<br/>config: prefix]
        CLCS[CatalogueConfigStorage<br/>catcfg: prefix]
        TIS[TypeIndexStorage<br/>typeidx: prefix]
        CATIS[CategoryIndexStorage<br/>cat: prefix]
        CLIS[CatalogueIndexStorage<br/>catl: prefix]
        TXIS[TextIndexStorage<br/>idx: prefix]
    end
    
    subgraph High-Level Entity Facades
        ES[EntityStore<br/>metadata + type index]
        DS[DocumentStore<br/>properties]
        CNS[ContainerStore<br/>children]
        CAS[CategoryStore<br/>config + index + children]
        CLS[CatalogueStore<br/>config + index]
        IDS[IndexStore<br/>config + text index]
    end
    
    subgraph Engine
        E[EmbeddedEngine]
    end
    
    EMS --> ES
    TIS --> ES
    PS --> DS
    CS --> CNS
    CCS --> CAS
    CATIS --> CAS
    CS --> CAS
    CLCS --> CLS
    CLIS --> CLS
    CCS --> IDS
    TXIS --> IDS
    
    E --> ES
    E --> DS
    E --> CNS
    E --> CAS
    E --> CLS
    E --> IDS
```

## Low-Level Prefix Stores

Each prefix store handles exactly one key prefix and provides type-safe operations.

**Naming Convention:** Prefix stores use the `Storage` suffix (e.g., `EntityMetadataStorage`).

### EntityMetadataStorage
**Prefix:** `entity:`

**Key Format:** `entity:<path>`

**Value:** Serialized `(EntityType type, string? type_label)`

```vala
public class EntityMetadataStorage : Object {
    public EntityMetadataStorage(Dbm dbm);
    
    public void store_metadata(EntityPath path, EntityType type, string? type_label) throws StorageError;
    public EntityType? get_type(EntityPath path) throws StorageError;
    public string? get_type_label(EntityPath path) throws StorageError;
    public bool exists(EntityPath path);
    public void delete(EntityPath path) throws StorageError;
}
```

### PropertiesStorage
**Prefix:** `props:`

**Key Format:** `props:<path>`

**Value:** Serialized `Properties` dictionary

```vala
public class PropertiesStorage : Object {
    public PropertiesStorage(Dbm dbm);
    
    public void store(EntityPath path, Properties properties) throws StorageError;
    public Properties? load(EntityPath path) throws StorageError;
    public void delete(EntityPath path) throws StorageError;
}
```

### ChildrenStorage
**Prefix:** `children:`

**Key Format:** `children:<parent_path>`

**Value:** Serialized array of child names

```vala
public class ChildrenStorage : Object {
    public ChildrenStorage(Dbm dbm);
    
    public void add_child(EntityPath parent, string child_name) throws StorageError;
    public void remove_child(EntityPath parent, string child_name) throws StorageError;
    public bool has_child(EntityPath parent, string child_name) throws StorageError;
    public Enumerable<string> get_children(EntityPath parent) throws StorageError;
    public void delete(EntityPath parent) throws StorageError;
}
```

### CategoryConfigStorage
**Prefix:** `config:`

**Key Format:** `config:<path>`

**Value:** Serialized `(string type_label, string expression)`

```vala
public class CategoryConfigStorage : Object {
    public CategoryConfigStorage(Dbm dbm);
    
    public void store(EntityPath path, string type_label, string expression) throws StorageError;
    public CategoryConfig? load(EntityPath path) throws StorageError;
    public void delete(EntityPath path) throws StorageError;
}

public class CategoryConfig : Object {
    public string type_label { get; construct set; }
    public string expression { get; construct set; }
}
```

### CatalogueConfigStorage
**Prefix:** `catcfg:`

**Key Format:** `catcfg:<path>`

**Value:** Serialized `(string type_label, string expression)`

```vala
public class CatalogueConfigStorage : Object {
    public CatalogueConfigStorage(Dbm dbm);
    
    public void store(EntityPath path, string type_label, string expression) throws StorageError;
    public CatalogueConfig? load(EntityPath path) throws StorageError;
    public void delete(EntityPath path) throws StorageError;
}

public class CatalogueConfig : Object {
    public string type_label { get; construct set; }
    public string expression { get; construct set; }
}
```

### TypeIndexStorage
**Prefix:** `typeidx:`

**Key Format:** `typeidx:<type_label>`

**Value:** Serialized array of document paths

```vala
public class TypeIndexStorage : Object {
    public TypeIndexStorage(Dbm dbm);
    
    public void add_document(string type_label, string doc_path) throws StorageError;
    public void remove_document(string type_label, string doc_path) throws StorageError;
    public Enumerable<string> get_documents(string type_label);
}
```

### CategoryIndexStorage
**Prefix:** `cat:`

**Key Format:** `cat:<category_path>:members`

**Value:** Serialized array of document paths

```vala
public class CategoryIndexStorage : Object {
    public CategoryIndexStorage(Dbm dbm);
    
    // Single member operations
    public void add_member(string category_path, string doc_path) throws StorageError;
    public void remove_member(string category_path, string doc_path) throws StorageError;
    
    // Batch member operations
    public void add_members(string category_path, Enumerable<string> doc_paths) throws StorageError;
    public void remove_members(string category_path, Enumerable<string> doc_paths) throws StorageError;
    public void set_members(string category_path, Enumerable<string> doc_paths) throws StorageError;
    
    // Query and lifecycle
    public Enumerable<string> get_members(string category_path);
    public void clear(string category_path) throws StorageError;
}
```

### CatalogueIndexStorage
**Prefix:** `catl:`

**Key Formats:**
- `catl:<catalogue_path>:keys` - List of group keys
- `catl:<catalogue_path>:group:<key>` - Document paths in group

```vala
public class CatalogueIndexStorage : Object {
    public CatalogueIndexStorage(Dbm dbm);
    
    // Group operations
    public void add_to_group(string catalogue_path, string key, string doc_path) throws StorageError;
    public void remove_from_group(string catalogue_path, string key, string doc_path) throws StorageError;
    public Enumerable<string> get_group_members(string catalogue_path, string key);
    
    // Key operations
    public void add_key(string catalogue_path, string key) throws StorageError;
    public void remove_key(string catalogue_path, string key) throws StorageError;
    public Enumerable<string> get_keys(string catalogue_path);
    
    // Clear all
    public void clear(string catalogue_path) throws StorageError;
}
```

### TextIndexStorage
**Prefix:** `idx:`

**Key Formats:**
- `idx:<index_path>:tri:<trigram>` - Document paths containing trigram
- `idx:<index_path>:bi:<bigram>` - Trigrams containing bigram
- `idx:<index_path>:uni:<unigram>` - Bigrams starting with unigram
- `idx:<index_path>:doc:<doc_path>` - Cached document content

```vala
public class TextIndexStorage : Object {
    public TextIndexStorage(Dbm dbm);
    
    // Trigram index
    public void add_trigram(string index_path, string trigram, string doc_path) throws StorageError;
    public void remove_trigram(string index_path, string trigram, string doc_path) throws StorageError;
    public Enumerable<string> get_documents_for_trigram(string index_path, string trigram);
    
    // Bigram reverse index
    public void add_bigram_mapping(string index_path, string bigram, string trigram) throws StorageError;
    public Enumerable<string> get_trigrams_for_bigram(string index_path, string bigram);
    
    // Unigram reverse index
    public void add_unigram_mapping(string index_path, string unigram, string bigram) throws StorageError;
    public Enumerable<string> get_bigrams_for_unigram(string index_path, string unigram);
    
    // Document content cache
    public void store_document_content(string index_path, string doc_path, string content) throws StorageError;
    public string? get_document_content(string index_path, string doc_path);
    public void remove_document_content(string index_path, string doc_path) throws StorageError;
    
    // Clear all
    public void clear(string index_path) throws StorageError;
}
```

## High-Level Entity Facades

Facades compose prefix stores to provide entity-specific APIs.

**Naming Convention:** Entity facades use the `Store` suffix (e.g., `EntityStore`).

### EntityStore
**Composition:** `EntityMetadataStorage` + `TypeIndexStorage`

```vala
public class EntityStore : Object {
    public EntityStore(Dbm dbm);
    
    // Metadata operations
    public void store_metadata(EntityPath path, EntityType type, string? type_label) throws StorageError;
    public EntityType? get_type(EntityPath path) throws StorageError;
    public string? get_type_label(EntityPath path) throws StorageError;
    public bool exists(EntityPath path);
    public void delete(EntityPath path) throws StorageError;
    
    // Type index operations
    public void register_document_type(string type_label, string doc_path) throws StorageError;
    public void unregister_document_type(string type_label, string doc_path) throws StorageError;
    public Enumerable<string> get_documents_by_type(string type_label);
}
```

### DocumentStore
**Composition:** `PropertiesStorage`

```vala
public class DocumentStore : Object {
    public DocumentStore(Dbm dbm);
    
    public void store_properties(EntityPath path, Properties properties) throws StorageError;
    public Properties? load_properties(EntityPath path) throws StorageError;
    public void delete(EntityPath path) throws StorageError;
}
```

### ContainerStore
**Composition:** `ChildrenStorage`

```vala
public class ContainerStore : Object {
    public ContainerStore(Dbm dbm);
    
    public void add_child(EntityPath parent, string child_name) throws StorageError;
    public void remove_child(EntityPath parent, string child_name) throws StorageError;
    public bool has_child(EntityPath parent, string child_name) throws StorageError;
    public Enumerable<string> get_children(EntityPath parent) throws StorageError;
}
```

### CategoryStore
**Composition:** `CategoryConfigStorage` + `CategoryIndexStorage` + `ChildrenStorage`

```vala
public class CategoryStore : Object {
    public CategoryStore(Dbm dbm);
    
    // Configuration
    public void store_config(EntityPath path, string type_label, string expression) throws StorageError;
    public CategoryConfig? load_config(EntityPath path) throws StorageError;
    
    // Single member operations
    public void add_member(EntityPath category_path, string doc_path) throws StorageError;
    public void remove_member(EntityPath category_path, string doc_path) throws StorageError;
    
    // Batch member operations
    public void add_members(EntityPath category_path, Enumerable<string> doc_paths) throws StorageError;
    public void remove_members(EntityPath category_path, Enumerable<string> doc_paths) throws StorageError;
    public void set_members(EntityPath category_path, Enumerable<string> doc_paths) throws StorageError;
    public Enumerable<string> get_members(EntityPath category_path);
    
    // Structural children (for when entities are created inside category)
    public void add_child(EntityPath parent, string child_name) throws StorageError;
    public void remove_child(EntityPath parent, string child_name) throws StorageError;
    public Enumerable<string> get_children(EntityPath parent) throws StorageError;
    
    // Lifecycle
    public void delete(EntityPath path) throws StorageError;
}
```

### CatalogueStore
**Composition:** `CatalogueConfigStorage` + `CatalogueIndexStorage`

```vala
public class CatalogueStore : Object {
    public CatalogueStore(Dbm dbm);
    
    // Configuration
    public void store_config(EntityPath path, string type_label, string expression) throws StorageError;
    public CatalogueConfig? load_config(EntityPath path) throws StorageError;
    
    // Group operations
    public void add_to_group(EntityPath catalogue_path, string key, string doc_path) throws StorageError;
    public void remove_from_group(EntityPath catalogue_path, string key, string doc_path) throws StorageError;
    public Enumerable<string> get_group_members(EntityPath catalogue_path, string key);
    public Enumerable<string> get_group_keys(EntityPath catalogue_path);
    
    // Lifecycle
    public void delete(EntityPath path) throws StorageError;
}
```

### IndexStore
**Composition:** `CategoryConfigStorage` + `TextIndexStorage`

```vala
public class IndexStore : Object {
    public IndexStore(Dbm dbm);
    
    // Configuration (reuses CategoryConfigStorage with config: prefix)
    public void store_config(EntityPath path, string type_label, string expression) throws StorageError;
    public CategoryConfig? load_config(EntityPath path) throws StorageError;
    
    // Trigram index
    public void add_trigram(EntityPath index_path, string trigram, string doc_path) throws StorageError;
    public void remove_trigram(EntityPath index_path, string trigram, string doc_path) throws StorageError;
    public Enumerable<string> get_documents_for_trigram(EntityPath index_path, string trigram);
    
    // Reverse indices
    public void add_bigram_mapping(EntityPath index_path, string bigram, string trigram) throws StorageError;
    public Enumerable<string> get_trigrams_for_bigram(EntityPath index_path, string bigram);
    public void add_unigram_mapping(EntityPath index_path, string unigram, string bigram) throws StorageError;
    public Enumerable<string> get_bigrams_for_unigram(EntityPath index_path, string unigram);
    
    // Content cache
    public void store_document_content(EntityPath index_path, string doc_path, string content) throws StorageError;
    public string? get_document_content(EntityPath index_path, string doc_path);
    public void remove_document_content(EntityPath index_path, string doc_path) throws StorageError;
    
    // Lifecycle
    public void delete(EntityPath path) throws StorageError;
}
```

## Engine Integration

The `EmbeddedEngine` holds references to all entity facades:

```vala
public class EmbeddedEngine : Object, Core.Engine {
    private EntityStore _entity_store;
    private DocumentStore _document_store;
    private ContainerStore _container_store;
    private CategoryStore _category_store;
    private CatalogueStore _catalogue_store;
    private IndexStore _index_store;
    
    public EmbeddedEngine.with_path(string storage_path) {
        var dbm = new FilesystemDbm(storage_path);
        
        _entity_store = new EntityStore(dbm);
        _document_store = new DocumentStore(dbm);
        _container_store = new ContainerStore(dbm);
        _category_store = new CategoryStore(dbm);
        _catalogue_store = new CatalogueStore(dbm);
        _index_store = new IndexStore(dbm);
        
        // ... rest of initialization
    }
    
    // Public access for entity classes
    public EntityStore entity_store { get { return _entity_store; } }
    public DocumentStore document_store { get { return _document_store; } }
    public ContainerStore container_store { get { return _container_store; } }
    public CategoryStore category_store { get { return _category_store; } }
    public CatalogueStore catalogue_store { get { return _catalogue_store; } }
    public IndexStore index_store { get { return _index_store; } }
}
```

## Key Schema Summary

| Prefix | Storage Class | Description |
|--------|---------------|-------------|
| `entity:` | EntityMetadataStorage | Entity type and type_label |
| `props:` | PropertiesStorage | Document properties |
| `children:` | ChildrenStorage | Structural child names |
| `config:` | CategoryConfigStorage | Category/Index configuration |
| `catcfg:` | CatalogueConfigStorage | Catalogue configuration |
| `typeidx:` | TypeIndexStorage | Global type → documents index |
| `cat:` | CategoryIndexStorage | Category members index |
| `catl:` | CatalogueIndexStorage | Catalogue groups and keys |
| `idx:` | TextIndexStorage | N-gram indices and content cache |

## Implementation Strategy

Since this is a greenfields project, we can implement directly without backward compatibility concerns.

### Phase 1: Create Prefix Storage Classes
1. Create `EntityMetadataStorage` in `src/Storage/EntityMetadataStorage.vala`
2. Create `PropertiesStorage` in `src/Storage/PropertiesStorage.vala`
3. Create `ChildrenStorage` in `src/Storage/ChildrenStorage.vala`
4. Create `CategoryConfigStorage` in `src/Storage/CategoryConfigStorage.vala`
5. Create `CatalogueConfigStorage` in `src/Storage/CatalogueConfigStorage.vala`
6. Create `TypeIndexStorage` in `src/Storage/TypeIndexStorage.vala`
7. Create `CategoryIndexStorage` in `src/Storage/CategoryIndexStorage.vala`
8. Create `CatalogueIndexStorage` in `src/Storage/CatalogueIndexStorage.vala`
9. Create `TextIndexStorage` in `src/Storage/TextIndexStorage.vala`

### Phase 2: Create Entity Facade Classes
1. Create `EntityStore` in `src/Storage/EntityStore.vala`
2. Create `DocumentStore` in `src/Storage/DocumentStore.vala`
3. Create `ContainerStore` in `src/Storage/ContainerStore.vala`
4. Create `CategoryStore` in `src/Storage/CategoryStore.vala`
5. Create `CatalogueStore` in `src/Storage/CatalogueStore.vala`
6. Create `IndexStore` in `src/Storage/IndexStore.vala`

### Phase 3: Update EmbeddedEngine
1. Add `with_write_transaction()` helper method
2. Add facade references for all stores
3. Remove old `Storage` and `IndexManager` references

### Phase 4: Update Entity Classes
1. Update `Container.vala` to use `ContainerStore` and `EntityStore`
2. Update `Document.vala` to use `DocumentStore` and `EntityStore`
3. Update `Category.vala` to use `CategoryStore` and `EntityStore`
4. Update `Catalogue.vala` to use `CatalogueStore` and `EntityStore`
5. Update `Index.vala` to use `IndexStore` and `EntityStore`

### Phase 5: Remove Old Classes
1. Delete `src/Storage/Storage.vala`
2. Delete `src/Storage/IndexManager.vala`

### Phase 6: Update Tests
1. Update `tests/Storage/StorageTest.vala` to test new storage classes
2. Add tests for each prefix storage class
3. Add tests for each entity facade class

## Benefits of New Design

1. **Clear Separation**: Each prefix store has one responsibility
2. **Composable**: Facades compose stores as needed
3. **Testable**: Small, focused classes are easier to test
4. **Discoverable**: `engine.category_store.add_member()` is self-documenting
5. **Flexible**: Can use low-level stores directly if needed
6. **No Broken Abstractions**: No casting to concrete types
7. **Clear Key Schema**: All prefixes documented in one place

## Transaction Model

The new architecture uses a **transaction-per-write-request** model:

### Behavior

| Operation Type | Transaction Behavior |
|----------------|---------------------|
| **Write operations** | Automatically wrapped in a transaction |
| **Read operations** | No transaction (no overhead) |
| **Hooks** | Run within the same transaction as the triggering write |

### Benefits

1. **Atomicity where needed**: Create document + add to container = one transaction
2. **Hooks are atomic**: Update document + category membership updates = one transaction
3. **No explicit transaction management**: Caller doesn't need to think about transactions
4. **Consistent behavior**: Same model works for both EmbeddedEngine and RemoteEngine
5. **No read overhead**: Reads don't pay transaction cost

### Implementation

#### EmbeddedEngine

Write operations on entities wrap their work in a transaction:

```vala
// In Container.create_child()
public Document create_child(string name) throws EntityError {
    Document? doc = null;
    _engine.with_write_transaction(() => {
        // Create entity metadata
        _engine.entity_store.store_metadata(path.child(name), EntityType.DOCUMENT, type_label);
        
        // Add to container's children
        _engine.container_store.add_child(path, name);
        
        // Create document entity
        doc = new Document(_engine, path.child(name));
        
        // Hooks run within same transaction
        _engine.hooks.run_after_create(doc);
    });
    return doc;
}
```

The Engine provides a helper method:

```vala
public class EmbeddedEngine : Object, Core.Engine {
    private Dbm _dbm;
    
    public void with_write_transaction(WriteTransactionDelegate delegate) throws Error {
        _dbm.with_transaction(() => delegate());
    }
}
```

#### RemoteEngine (Server Side)

The server wraps each write request in a transaction:

```vala
// In ClientHandler
void handle_create_document(Message request) {
    _engine.with_write_transaction(() => {
        // Process the entire request in one transaction
        var path = request.get_path();
        var type_label = request.get_type_label();
        
        _engine.entity_store.store_metadata(path, EntityType.DOCUMENT, type_label);
        _engine.container_store.add_child(path.parent, path.name);
        
        // Hooks run within same transaction
        var doc = new Document(_engine, path);
        _engine.hooks.run_after_create(doc);
    });
}
```

### Examples

#### Creating a Document

```vala
// Single transaction covers:
// 1. Create entity metadata
// 2. Add to container's children list
// 3. Store initial properties
// 4. Run after_create hooks (e.g., add to categories)
var doc = container.create_child("my-doc");
```

#### Updating a Document

```vala
// Single transaction covers:
// 1. Update properties
// 2. Run after_update hooks (e.g., update category memberships, update indices)
doc.set_property("status", "active");
```

#### Deleting a Document

```vala
// Single transaction covers:
// 1. Remove from container's children list
// 2. Delete entity metadata
// 3. Delete properties
// 4. Run after_delete hooks (e.g., remove from categories, remove from indices)
doc.delete();
```

## Decisions Made

1. **Caching**: Deferred - no caching layer needed in the initial implementation. Can be added later if performance profiling indicates it's needed.

2. **Batch Operations**: Added `add_members()` and `remove_members()` batch methods to `CategoryIndexStorage` and `CategoryStore` for efficient bulk updates during reindexing operations.

3. **Transaction Model**: Transaction-per-write-request model selected. Write operations are automatically wrapped in transactions, reads have no transaction overhead, and hooks run within the same transaction as the triggering write.