# Storage Backends

Implexus provides a flexible storage layer through the [`Dbm`](src/Storage/Dbm.vala) interface, allowing applications to choose the most appropriate backend for their use case.

## Overview

The [`Dbm`](src/Storage/Dbm.vala:12) interface defines a low-level key-value storage abstraction with support for:

- Basic CRUD operations (`get`, `set`, `delete`, `has_key`)
- Key enumeration via the `keys` property
- Transaction support (`begin_transaction`, `commit_transaction`, `rollback_transaction`)

The storage layer is used by [`BasicStorage`](src/Storage/Storage.vala:186) to provide high-level entity persistence operations, and by [`IndexManager`](src/Storage/IndexManager.vala:49) for index storage.

## Available Backends

### FilesystemDbm

**Location:** [`src/Storage/FilesystemDbm.vala`](src/Storage/FilesystemDbm.vala:17)

A simple file-based storage implementation where each key-value pair is stored in a separate file within a directory.

#### Characteristics

| Property | Value |
|----------|-------|
| **Transaction Support** | Software-based (in-memory buffering) |
| **Performance** | Good for small datasets, degrades with size |
| **Dependencies** | None (pure GLib) |
| **File Format** | One file per key, hex-encoded filenames |

#### Best For

- Development and testing environments
- Small embedded applications
- Scenarios where external database dependencies are undesirable

#### Limitations

- Not suitable for high-volume production use
- Performance degrades significantly with large datasets due to filesystem overhead
- No compression or optimization for storage space

#### Usage

```vala
var dbm = new FilesystemDbm("/path/to/data/directory");
var storage = new BasicStorage(dbm);
```

---

### GdbmDbm

**Location:** [`src/Storage/Gdbm/GdbmDbm.vala`](src/Storage/Gdbm/GdbmDbm.vala:16)

A production-ready backend using the GNU DBM library for persistent key-value storage.

#### Characteristics

| Property | Value |
|----------|-------|
| **Transaction Support** | Software-based (in-memory buffering) |
| **Performance** | Good for medium datasets |
| **Dependencies** | libgdbm |
| **File Format** | Single database file with hash table structure |

#### Best For

- Single-threaded applications
- Medium workloads with moderate read/write ratios
- Applications already using GDBM in their stack

#### Limitations

- Limited concurrent access support (single writer recommended)
- Software-based transactions (not ACID)
- Database file can become fragmented over time

#### Usage

```vala
var dbm = new GdbmDbm();
dbm.open("/path/to/database.gdbm", false); // false = read-write mode
var storage = new BasicStorage(dbm);
```

---

### LmdbDbm

**Location:** [`src/Storage/Lmdb/LmdbDbm.vala`](src/Storage/Lmdb/LmdbDbm.vala:16)

A high-performance backend using the Lightning Memory-Mapped Database (LMDB).

#### Characteristics

| Property | Value |
|----------|-------|
| **Transaction Support** | Native ACID transactions |
| **Performance** | Excellent for read-heavy workloads |
| **Dependencies** | liblmdb |
| **File Format** | Memory-mapped B+tree |
| **Map Size** | Default 1GB (configurable) |

#### Features

- **Memory-mapped**: Direct OS-level caching for optimal read performance
- **B+tree structure**: Efficient range queries and ordered traversal
- **MVCC**: Multi-version concurrency control for consistent reads
- **ACID compliant**: Full transaction support with crash recovery

#### Best For

- High-performance applications
- Read-heavy workloads
- Multi-threaded read scenarios
- Applications requiring ACID guarantees

#### Limitations

- Fixed map size (must be configured at environment creation)
- Single writer at a time (multiple readers allowed)
- Write performance can be slower than reads due to copy-on-write

#### Usage

```vala
var dbm = new LmdbDbm();
dbm.open("/path/to/lmdb/directory", false); // false = read-write mode
var storage = new BasicStorage(dbm);
```

## Comparison Table

| Backend | Transaction Type | Read Performance | Write Performance | Concurrent Reads | Concurrent Writes | Dependencies | Recommended Use Case |
|---------|------------------|------------------|-------------------|------------------|-------------------|--------------|---------------------|
| FilesystemDbm | Software | Poor | Poor | No | No | None | Development, testing |
| GdbmDbm | Software | Good | Good | Limited | No | libgdbm | Single-threaded apps |
| LmdbDbm | Native ACID | Excellent | Good | Yes | Single writer | liblmdb | Production, high-performance |

## Configuration

When using [`EngineConfiguration`](src/Engine/EngineConfiguration.vala:60) for embedded mode, the storage backend is selected based on the application's needs:

```vala
// Create embedded configuration with storage path
var config = EngineConfiguration.embedded("/path/to/database");

// The EngineFactory creates the appropriate backend
var engine = EngineFactory.create(config);
```

### Custom Backend Selection

For direct control over the storage backend:

```vala
// Using LMDB for high performance
var lmdb = new LmdbDbm();
lmdb.open("/data/implexus-lmdb", false);
var storage = new BasicStorage(lmdb);
var index_manager = new IndexManager(lmdb);

// Using GDBM for medium workloads
var gdbm = new GdbmDbm();
gdbm.open("/data/implexus.gdbm", false);
var storage = new BasicStorage(gdbm);

// Using FilesystemDbm for testing
var fsdbm = new FilesystemDbm("/data/test-storage");
var storage = new BasicStorage(fsdbm);
```

## Transaction Behavior

All backends implement the same transaction interface, but with different guarantees:

### Software-Based Transactions (FilesystemDbm, GdbmDbm)

```vala
dbm.begin_transaction();
try {
    dbm.set("key1", value1);
    dbm.set("key2", value2);
    dbm.delete("key3");
    dbm.commit_transaction();
} catch (StorageError e) {
    dbm.rollback_transaction();
}
```

Operations are buffered in memory and applied atomically on commit. Rollback discards the buffer without modifying persistent storage.

### Native Transactions (LmdbDbm)

LMDB provides true ACID transactions with crash recovery and durability guarantees. The same API is used:

```vala
dbm.begin_transaction();
try {
    dbm.set("key1", value1);
    dbm.set("key2", value2);
    dbm.commit_transaction();
} catch (StorageError e) {
    dbm.rollback_transaction();
}
```

## Performance Considerations

### Read-Heavy Workloads

For applications with high read-to-write ratios, **LmdbDbm** is recommended due to:
- Memory-mapped files enable OS-level caching
- MVCC allows concurrent readers without blocking
- B+tree structure optimizes sequential access

### Write-Heavy Workloads

For write-intensive applications:
- **GdbmDbm** may perform better for single-writer scenarios
- **LmdbDbm** provides ACID guarantees but with copy-on-write overhead

### Small Datasets

For development or small embedded applications:
- **FilesystemDbm** is simplest with no external dependencies
- Performance is acceptable for datasets under a few thousand keys

## Migration Between Backends

To migrate data between backends:

```vala
// Open source backend
var source = new GdbmDbm();
source.open("/old/database.gdbm", true); // read-only

// Open target backend
var target = new LmdbDbm();
target.open("/new/lmdb/dir", false); // read-write

// Copy all keys
foreach (var key in source.keys) {
    var value = source.get(key);
    if (value != null) {
        target.set(key, (!) value);
    }
}
```

## See Also

- [KEY-SCHEMA.md](KEY-SCHEMA.md) - Key schema documentation
- [Architecture/07-Storage-Layer.md](Architecture/07-Storage-Layer.md) - Storage layer architecture