Implexus/STORAGE-BACKENDS.md

Storage Backends

Implexus provides a flexible storage layer through the Dbm interface, allowing applications to choose the most appropriate backend for their use case.

Overview

The Dbm interface defines a low-level key-value storage abstraction with support for:

• Basic CRUD operations (get, set, delete, has_key)
• Key enumeration via the keys property
• Transaction support (begin_transaction, commit_transaction, rollback_transaction)

The storage layer is used by BasicStorage to provide high-level entity persistence operations, and by IndexManager for index storage.

Available Backends

FilesystemDbm

Location: src/Storage/FilesystemDbm.vala

A simple file-based storage implementation where each key-value pair is stored in a separate file within a directory.

Characteristics

Property	Value
Transaction Support	Software-based (in-memory buffering)
Performance	Good for small datasets, degrades with size
Dependencies	None (pure GLib)
File Format	One file per key, hex-encoded filenames

Best For

• Development and testing environments
• Small embedded applications
• Scenarios where external database dependencies are undesirable

Limitations

• Not suitable for high-volume production use
• Performance degrades significantly with large datasets due to filesystem overhead
• No compression or optimization for storage space

Usage

var dbm = new FilesystemDbm("/path/to/data/directory");
var storage = new BasicStorage(dbm);

---

GdbmDbm

Location: src/Storage/Gdbm/GdbmDbm.vala

A production-ready backend using the GNU DBM library for persistent key-value storage.

Characteristics

Property	Value
Transaction Support	Software-based (in-memory buffering)
Performance	Good for medium datasets
Dependencies	libgdbm
File Format	Single database file with hash table structure

Best For

• Single-threaded applications
• Medium workloads with moderate read/write ratios
• Applications already using GDBM in their stack

Limitations

• Limited concurrent access support (single writer recommended)
• Software-based transactions (not ACID)
• Database file can become fragmented over time

Usage

var dbm = new GdbmDbm();
dbm.open("/path/to/database.gdbm", false); // false = read-write mode
var storage = new BasicStorage(dbm);

---

LmdbDbm

Location: src/Storage/Lmdb/LmdbDbm.vala

A high-performance backend using the Lightning Memory-Mapped Database (LMDB).

Characteristics

Property	Value
Transaction Support	Native ACID transactions
Performance	Excellent for read-heavy workloads
Dependencies	liblmdb
File Format	Memory-mapped B+tree
Map Size	Default 1GB (configurable)

Features

• Memory-mapped: Direct OS-level caching for optimal read performance
• B+tree structure: Efficient range queries and ordered traversal
• MVCC: Multi-version concurrency control for consistent reads
• ACID compliant: Full transaction support with crash recovery

Best For

• High-performance applications
• Read-heavy workloads
• Multi-threaded read scenarios
• Applications requiring ACID guarantees

Limitations

• Fixed map size (must be configured at environment creation)
• Single writer at a time (multiple readers allowed)
• Write performance can be slower than reads due to copy-on-write

Usage

var dbm = new LmdbDbm();
dbm.open("/path/to/lmdb/directory", false); // false = read-write mode
var storage = new BasicStorage(dbm);

Comparison Table

Backend	Transaction Type	Read Performance	Write Performance	Concurrent Reads	Concurrent Writes	Dependencies	Recommended Use Case
FilesystemDbm	Software	Poor	Poor	No	No	None	Development, testing
GdbmDbm	Software	Good	Good	Limited	No	libgdbm	Single-threaded apps
LmdbDbm	Native ACID	Excellent	Good	Yes	Single writer	liblmdb	Production, high-performance

Configuration

When using EngineConfiguration for embedded mode, the storage backend is selected based on the application's needs:

// Create embedded configuration with storage path
var config = EngineConfiguration.embedded("/path/to/database");

// The EngineFactory creates the appropriate backend
var engine = EngineFactory.create(config);

Custom Backend Selection

For direct control over the storage backend:

// Using LMDB for high performance
var lmdb = new LmdbDbm();
lmdb.open("/data/implexus-lmdb", false);
var storage = new BasicStorage(lmdb);
var index_manager = new IndexManager(lmdb);

// Using GDBM for medium workloads
var gdbm = new GdbmDbm();
gdbm.open("/data/implexus.gdbm", false);
var storage = new BasicStorage(gdbm);

// Using FilesystemDbm for testing
var fsdbm = new FilesystemDbm("/data/test-storage");
var storage = new BasicStorage(fsdbm);

Transaction Behavior

All backends implement the same transaction interface, but with different guarantees:

Software-Based Transactions (FilesystemDbm, GdbmDbm)

dbm.begin_transaction();
try {
    dbm.set("key1", value1);
    dbm.set("key2", value2);
    dbm.delete("key3");
    dbm.commit_transaction();
} catch (StorageError e) {
    dbm.rollback_transaction();
}

Operations are buffered in memory and applied atomically on commit. Rollback discards the buffer without modifying persistent storage.

Native Transactions (LmdbDbm)

LMDB provides true ACID transactions with crash recovery and durability guarantees. The same API is used:

dbm.begin_transaction();
try {
    dbm.set("key1", value1);
    dbm.set("key2", value2);
    dbm.commit_transaction();
} catch (StorageError e) {
    dbm.rollback_transaction();
}

Performance Considerations

Read-Heavy Workloads

For applications with high read-to-write ratios, LmdbDbm is recommended due to:

• Memory-mapped files enable OS-level caching
• MVCC allows concurrent readers without blocking
• B+tree structure optimizes sequential access

Write-Heavy Workloads

For write-intensive applications:

• GdbmDbm may perform better for single-writer scenarios
• LmdbDbm provides ACID guarantees but with copy-on-write overhead

Small Datasets

For development or small embedded applications:

• FilesystemDbm is simplest with no external dependencies
• Performance is acceptable for datasets under a few thousand keys

Migration Between Backends

To migrate data between backends:

// Open source backend
var source = new GdbmDbm();
source.open("/old/database.gdbm", true); // read-only

// Open target backend
var target = new LmdbDbm();
target.open("/new/lmdb/dir", false); // read-write

// Copy all keys
foreach (var key in source.keys) {
    var value = source.get(key);
    if (value != null) {
        target.set(key, (!) value);
    }
}

See Also

• KEY-SCHEMA.md - Key schema documentation
• Architecture/07-Storage-Layer.md - Storage layer architecture