elsa-core/specs/005-structured-log-persistence/spec.md

Feature Specification: Structured Log Persistence

Feature Branch: 005-structured-log-persistence
Created: 2026-05-12
Status: Draft
Input: User description: "Make structured log storage a pluggable concern. Keep in-memory support, add an easy durable SQLite storage story first, use FluentMigrator for schema management, and defer OpenTelemetry/exporter scope."

Clarifications

Session 2026-05-12

• Q: Should this feature add vendor sinks such as Logstash, Datadog, Splunk, Loki, or Seq now? -> A: No. Keep the first slice small.
• Q: Should this feature add an OTLP logs exporter now? -> A: No. Defer OpenTelemetry until the diagnostics OpenTelemetry boundary is clearer.
• Q: What storage providers should exist initially? -> A: Preserve in-memory storage and add opt-in SQLite durable storage.
• Q: Should SQLite be implemented as a one-off store? -> A: No. Implement it as the first relational provider so SQL Server, PostgreSQL, MySQL, and similar providers can be added later.
• Q: Should EF Core be used for persistence? -> A: No. Avoid EF Core and per-provider EF migrations for this feature.
• Q: Should FluentMigrator manage schema creation and upgrades? -> A: Yes. Use FluentMigrator for schema versioning and migration execution; use explicit SQL/Dapper-style access for the hot storage path.

Session 2026-05-13

• Q: What durability guarantee should SQLite writes provide on the logging hot path? -> A: Async batched writes with graceful-shutdown flush; a crash may lose queued events.
• Q: What should happen when the SQLite write queue is full? -> A: Drop newest queued events and report dropped counts/metrics.
• Q: Should SQLite migrations run automatically on startup? -> A: Run migrations on startup by default, with opt-out.
• Q: How should SQLite persist timestamps? -> A: Store timestamps as UTC ISO-8601 text.
• Q: What default retention policy should SQLite use? -> A: No default deletion; retention runs only when max age or max rows is configured.

User Scenarios & Testing (mandatory)

User Story 1 - Preserve the existing in-memory structured logs behavior (Priority: P1)

An Elsa host developer enables structured logs without configuring durable persistence and gets the same bounded in-memory recent history and live streaming behavior as today.

Why this priority: The storage refactor must not regress the current module. In-memory remains the zero-configuration default and the fastest validation path.

Independent Test: Enable UseStructuredLogs with no persistence options, emit several ILogger events, and verify recent queries, filtering, source listing, live subscriptions, redaction, and dropped-event summaries continue to work.

Acceptance Scenarios:

1. Given no durable store is configured, When structured logs are enabled, Then the module uses bounded in-memory storage by default.
2. Given recent logs are queried through the existing REST endpoint, When events have been captured, Then the endpoint returns filtered recent events using the same contract as before.
3. Given Studio subscribes over SignalR, When new matching events are emitted, Then live events and dropped-event summaries continue to stream.

---

User Story 2 - Enable durable SQLite structured log storage (Priority: P2)

An Elsa host developer opts into SQLite persistence so structured log entries survive process restarts and can be queried by Studio after restart.

Why this priority: Customers need an easy persistence story without standing up a separate database service or observability stack.

Independent Test: Configure SQLite storage, emit structured log entries, restart or recreate the service provider with the same database file, and verify recent queries return the previously written events.

Acceptance Scenarios:

1. Given SQLite storage is configured with a database file, When structured log events are captured, Then the events are written durably to SQLite.
2. Given the host restarts with the same SQLite database, When recent structured logs are queried, Then persisted events are returned according to the filter.
3. Given retention settings are configured, When cleanup runs, Then old or excess records are deleted without breaking recent queries.
4. Given schema migrations have not run, When the SQLite provider starts, Then FluentMigrator creates or upgrades the structured log schema.
5. Given an operator disables startup migrations, When the SQLite provider starts, Then it does not run migrations and requires the schema to be prepared separately.
6. Given no SQLite retention settings are configured, When cleanup runs, Then no persisted log entries are deleted by default.

---

User Story 3 - Keep relational persistence extensible for future databases (Priority: P3)

An Elsa maintainer can add SQL Server, PostgreSQL, MySQL, or another relational provider later without rewriting the structured logs module or changing Studio contracts.

Why this priority: SQLite should prove the relational persistence model, not become a dead-end implementation.

Independent Test: Review the relational contracts, dialect hooks, migrations, and SQLite package to confirm provider-specific behavior is isolated from the core structured logs module.

Acceptance Scenarios:

1. Given a future database provider is added, When it supplies a connection factory, SQL dialect, and FluentMigrator runner registration, Then it can reuse the shared relational store implementation.
2. Given a provider needs database-specific DDL, When migrations run, Then the migration can branch by database while keeping a shared migration version.
3. Given Studio uses existing structured logs REST and SignalR contracts, When storage changes from in-memory to SQLite, Then Studio requires no API changes.

Edge Cases

• Durable writes must not block ILogger callers on slow disk I/O; SQLite persistence accepts possible loss of queued-but-unflushed events after process crashes.
• If the SQLite write queue reaches capacity, newest events are dropped and loss counts are reported instead of blocking logging calls or growing memory without bound.
• A host can emit logs before migrations complete if startup ordering is wrong.
• Multiple app instances can start against a shared future relational database and attempt migrations concurrently.
• SQLite file paths can be missing, relative, or point to directories without write permissions.
• A process can crash after events enter a background queue but before they are flushed.
• Free-text filtering over JSON fields can be provider-specific and initially approximate.
• Retention cleanup can race with recent queries or live subscriptions.
• Schema changes must preserve already persisted log events where practical.

Requirements (mandatory)

Functional Requirements

Storage abstraction

• FR-001: The structured logs module MUST separate queryable storage from live streaming and capture/provider facade responsibilities.
• FR-002: The existing IStructuredLogProvider contract MAY remain as the facade used by REST endpoints and SignalR, but storage-specific responsibilities MUST be represented by a replaceable store abstraction.
• FR-003: The storage abstraction MUST support appending redacted StructuredLogEvent values, querying recent events by StructuredLogFilter, and listing structured log sources.
• FR-004: Live streaming MUST remain available when durable storage is configured.
• FR-005: Redaction MUST continue to happen before any event reaches in-memory storage, SQLite storage, or live subscribers.

In-memory default

• FR-006: The module MUST keep bounded in-memory storage as the default when no durable storage provider is configured.
• FR-007: The in-memory implementation MUST preserve existing recent query, source listing, subscription, dropped-event summary, capacity, and filtering behavior.

SQLite durable storage

• FR-008: The feature MUST add an opt-in SQLite structured log persistence provider.
• FR-009: SQLite storage MUST persist structured log events across process restarts.
• FR-010: SQLite storage MUST support the existing StructuredLogFilter fields used by the REST recent-log endpoint.
• FR-011: SQLite storage MUST persist scalar filter fields as queryable columns and persist exception, scopes, and properties as serialized JSON.
• FR-012: SQLite storage MUST provide configurable retention by maximum age and/or maximum row count.
• FR-013: SQLite storage MUST persist Timestamp and ReceivedAt as UTC ISO-8601 text values.
• FR-014: SQLite writes MUST be batched or queued so logging calls do not synchronously wait on disk I/O.
• FR-015: SQLite storage MUST flush queued events during graceful shutdown where possible.
• FR-016: SQLite storage MAY lose events that were queued but not flushed before an ungraceful process crash.
• FR-017: SQLite storage MUST use a bounded write queue.
• FR-018: When the SQLite write queue is full, SQLite storage MUST drop newly received events instead of blocking logging calls or using unbounded memory.
• FR-019: SQLite storage MUST expose dropped-write counts through IStructuredLogWriteBuffer.DroppedWriteCount and log warning summaries.
• FR-020: SQLite retention MUST delete no records by default unless MaxAge, MaxRows, or both are configured.

Relational extensibility

• FR-021: Shared relational persistence code MUST be reusable by future SQL Server, PostgreSQL, MySQL, and similar providers.
• FR-022: Provider-specific SQL syntax MUST be isolated behind dialect or provider services.
• FR-023: The first SQLite provider MUST not introduce SQLite-only assumptions into the core structured logs module.
• FR-024: Future relational providers MUST be able to supply their own connection factory, dialect, and migration runner configuration without changing REST, SignalR, or Studio contracts.

Schema management

• FR-025: Relational schema creation and upgrades MUST use FluentMigrator.
• FR-026: Migrations MUST be versioned and idempotently executable by the host.
• FR-027: Provider packages MUST register only the FluentMigrator runner dependencies they need.
• FR-028: SQLite storage MUST run migrations on startup by default.
• FR-029: SQLite storage MUST provide an option to disable startup migrations for hosts that prepare schemas separately.
• FR-030: The feature MUST document startup migration behavior and the multi-instance locking consideration for future shared database providers.

Configuration and safety

• FR-031: Host configuration MUST make the active storage mode explicit when SQLite is selected.
• FR-032: The existing no-configuration in-memory setup MUST continue to work.
• FR-033: The feature MUST document that OpenTelemetry/exporter sinks are deferred and out of scope for this persistence slice.

Key Entities (include if feature involves data)

• Structured Log Store: Queryable storage for redacted structured log events and sources.
• Structured Log Live Feed: Runtime stream of new events and dropped-event summaries for SignalR subscribers.
• Relational Structured Log Record: Database representation of a StructuredLogEvent with indexed scalar columns and JSON payload columns.
• Relational Storage Dialect: Provider-specific SQL and parameter behavior needed by the shared relational store.
• Structured Log Migration Runner: FluentMigrator-based schema creation and upgrade service.
• Structured Log Retention Policy: Settings and cleanup behavior that bound durable storage growth.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Existing structured logs unit and integration tests continue to pass with the in-memory default.
• SC-002: New tests verify SQLite persistence survives service-provider or process recreation with the same database file.
• SC-003: New tests verify SQLite recent queries honor level, category, source, workflow, correlation, trace/span, time range, and limit filters.
• SC-004: New tests verify FluentMigrator creates the SQLite schema from an empty database.
• SC-005: New tests verify retention cleanup removes expired or excess records only when retention settings are configured.
• SC-006: New tests verify queued SQLite writes are flushed during graceful shutdown.
• SC-007: New tests verify a full SQLite write queue drops newest events and reports dropped-write counts.
• SC-008: New tests verify SQLite startup migrations run by default and can be disabled.
• SC-009: New tests verify SQLite stores and filters UTC ISO-8601 timestamp text values consistently.
• SC-010: New tests verify SQLite retention deletes no records by default.
• SC-011: Documentation shows both zero-configuration in-memory setup and opt-in SQLite setup.
• SC-012: The core structured logs module does not reference SQLite-specific types.

Assumptions

• The structured logs module from 004-diagnostics-structured-logs is available and remains the owning diagnostics module.
• SQLite is the only durable provider implemented in this feature.
• Future relational providers can be added in separate specs and packages.
• OpenTelemetry logs export, vendor sinks, raw console streaming, and trace/metric exploration remain out of scope.
• The implementation can introduce new package dependencies such as FluentMigrator and Dapper where appropriate, with versions managed centrally.