# Feature Specification: Structured Log Persistence **Feature Branch**: `005-structured-log-persistence` **Created**: 2026-05-12 **Status**: Draft **Input**: User description: "Make structured log storage a pluggable concern. Keep in-memory support, add an easy durable SQLite storage story first, use FluentMigrator for schema management, and defer OpenTelemetry/exporter scope." ## Clarifications ### Session 2026-05-12 - Q: Should this feature add vendor sinks such as Logstash, Datadog, Splunk, Loki, or Seq now? -> A: No. Keep the first slice small. - Q: Should this feature add an OTLP logs exporter now? -> A: No. Defer OpenTelemetry until the diagnostics OpenTelemetry boundary is clearer. - Q: What storage providers should exist initially? -> A: Preserve in-memory storage and add opt-in SQLite durable storage. - Q: Should SQLite be implemented as a one-off store? -> A: No. Implement it as the first relational provider so SQL Server, PostgreSQL, MySQL, and similar providers can be added later. - Q: Should EF Core be used for persistence? -> A: No. Avoid EF Core and per-provider EF migrations for this feature. - Q: Should FluentMigrator manage schema creation and upgrades? -> A: Yes. Use FluentMigrator for schema versioning and migration execution; use explicit SQL/Dapper-style access for the hot storage path. ### Session 2026-05-13 - Q: What durability guarantee should SQLite writes provide on the logging hot path? -> A: Async batched writes with graceful-shutdown flush; a crash may lose queued events. - Q: What should happen when the SQLite write queue is full? -> A: Drop newest queued events and report dropped counts/metrics. - Q: Should SQLite migrations run automatically on startup? -> A: Run migrations on startup by default, with opt-out. - Q: How should SQLite persist timestamps? -> A: Store timestamps as UTC ISO-8601 text. - Q: What default retention policy should SQLite use? -> A: No default deletion; retention runs only when max age or max rows is configured. ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Preserve the existing in-memory structured logs behavior (Priority: P1) An Elsa host developer enables structured logs without configuring durable persistence and gets the same bounded in-memory recent history and live streaming behavior as today. **Why this priority**: The storage refactor must not regress the current module. In-memory remains the zero-configuration default and the fastest validation path. **Independent Test**: Enable `UseStructuredLogs` with no persistence options, emit several `ILogger` events, and verify recent queries, filtering, source listing, live subscriptions, redaction, and dropped-event summaries continue to work. **Acceptance Scenarios**: 1. **Given** no durable store is configured, **When** structured logs are enabled, **Then** the module uses bounded in-memory storage by default. 2. **Given** recent logs are queried through the existing REST endpoint, **When** events have been captured, **Then** the endpoint returns filtered recent events using the same contract as before. 3. **Given** Studio subscribes over SignalR, **When** new matching events are emitted, **Then** live events and dropped-event summaries continue to stream. --- ### User Story 2 - Enable durable SQLite structured log storage (Priority: P2) An Elsa host developer opts into SQLite persistence so structured log entries survive process restarts and can be queried by Studio after restart. **Why this priority**: Customers need an easy persistence story without standing up a separate database service or observability stack. **Independent Test**: Configure SQLite storage, emit structured log entries, restart or recreate the service provider with the same database file, and verify recent queries return the previously written events. **Acceptance Scenarios**: 1. **Given** SQLite storage is configured with a database file, **When** structured log events are captured, **Then** the events are written durably to SQLite. 2. **Given** the host restarts with the same SQLite database, **When** recent structured logs are queried, **Then** persisted events are returned according to the filter. 3. **Given** retention settings are configured, **When** cleanup runs, **Then** old or excess records are deleted without breaking recent queries. 4. **Given** schema migrations have not run, **When** the SQLite provider starts, **Then** FluentMigrator creates or upgrades the structured log schema. 5. **Given** an operator disables startup migrations, **When** the SQLite provider starts, **Then** it does not run migrations and requires the schema to be prepared separately. 6. **Given** no SQLite retention settings are configured, **When** cleanup runs, **Then** no persisted log entries are deleted by default. --- ### User Story 3 - Keep relational persistence extensible for future databases (Priority: P3) An Elsa maintainer can add SQL Server, PostgreSQL, MySQL, or another relational provider later without rewriting the structured logs module or changing Studio contracts. **Why this priority**: SQLite should prove the relational persistence model, not become a dead-end implementation. **Independent Test**: Review the relational contracts, dialect hooks, migrations, and SQLite package to confirm provider-specific behavior is isolated from the core structured logs module. **Acceptance Scenarios**: 1. **Given** a future database provider is added, **When** it supplies a connection factory, SQL dialect, and FluentMigrator runner registration, **Then** it can reuse the shared relational store implementation. 2. **Given** a provider needs database-specific DDL, **When** migrations run, **Then** the migration can branch by database while keeping a shared migration version. 3. **Given** Studio uses existing structured logs REST and SignalR contracts, **When** storage changes from in-memory to SQLite, **Then** Studio requires no API changes. ### Edge Cases - Durable writes must not block `ILogger` callers on slow disk I/O; SQLite persistence accepts possible loss of queued-but-unflushed events after process crashes. - If the SQLite write queue reaches capacity, newest events are dropped and loss counts are reported instead of blocking logging calls or growing memory without bound. - A host can emit logs before migrations complete if startup ordering is wrong. - Multiple app instances can start against a shared future relational database and attempt migrations concurrently. - SQLite file paths can be missing, relative, or point to directories without write permissions. - A process can crash after events enter a background queue but before they are flushed. - Free-text filtering over JSON fields can be provider-specific and initially approximate. - Retention cleanup can race with recent queries or live subscriptions. - Schema changes must preserve already persisted log events where practical. ## Requirements *(mandatory)* ### Functional Requirements **Storage abstraction** - **FR-001**: The structured logs module MUST separate queryable storage from live streaming and capture/provider facade responsibilities. - **FR-002**: The existing `IStructuredLogProvider` contract MAY remain as the facade used by REST endpoints and SignalR, but storage-specific responsibilities MUST be represented by a replaceable store abstraction. - **FR-003**: The storage abstraction MUST support appending redacted `StructuredLogEvent` values, querying recent events by `StructuredLogFilter`, and listing structured log sources. - **FR-004**: Live streaming MUST remain available when durable storage is configured. - **FR-005**: Redaction MUST continue to happen before any event reaches in-memory storage, SQLite storage, or live subscribers. **In-memory default** - **FR-006**: The module MUST keep bounded in-memory storage as the default when no durable storage provider is configured. - **FR-007**: The in-memory implementation MUST preserve existing recent query, source listing, subscription, dropped-event summary, capacity, and filtering behavior. **SQLite durable storage** - **FR-008**: The feature MUST add an opt-in SQLite structured log persistence provider. - **FR-009**: SQLite storage MUST persist structured log events across process restarts. - **FR-010**: SQLite storage MUST support the existing `StructuredLogFilter` fields used by the REST recent-log endpoint. - **FR-011**: SQLite storage MUST persist scalar filter fields as queryable columns and persist exception, scopes, and properties as serialized JSON. - **FR-012**: SQLite storage MUST provide configurable retention by maximum age and/or maximum row count. - **FR-013**: SQLite storage MUST persist `Timestamp` and `ReceivedAt` as UTC ISO-8601 text values. - **FR-014**: SQLite writes MUST be batched or queued so logging calls do not synchronously wait on disk I/O. - **FR-015**: SQLite storage MUST flush queued events during graceful shutdown where possible. - **FR-016**: SQLite storage MAY lose events that were queued but not flushed before an ungraceful process crash. - **FR-017**: SQLite storage MUST use a bounded write queue. - **FR-018**: When the SQLite write queue is full, SQLite storage MUST drop newly received events instead of blocking logging calls or using unbounded memory. - **FR-019**: SQLite storage MUST expose dropped-write counts through `IStructuredLogWriteBuffer.DroppedWriteCount` and log warning summaries. - **FR-020**: SQLite retention MUST delete no records by default unless `MaxAge`, `MaxRows`, or both are configured. **Relational extensibility** - **FR-021**: Shared relational persistence code MUST be reusable by future SQL Server, PostgreSQL, MySQL, and similar providers. - **FR-022**: Provider-specific SQL syntax MUST be isolated behind dialect or provider services. - **FR-023**: The first SQLite provider MUST not introduce SQLite-only assumptions into the core structured logs module. - **FR-024**: Future relational providers MUST be able to supply their own connection factory, dialect, and migration runner configuration without changing REST, SignalR, or Studio contracts. **Schema management** - **FR-025**: Relational schema creation and upgrades MUST use FluentMigrator. - **FR-026**: Migrations MUST be versioned and idempotently executable by the host. - **FR-027**: Provider packages MUST register only the FluentMigrator runner dependencies they need. - **FR-028**: SQLite storage MUST run migrations on startup by default. - **FR-029**: SQLite storage MUST provide an option to disable startup migrations for hosts that prepare schemas separately. - **FR-030**: The feature MUST document startup migration behavior and the multi-instance locking consideration for future shared database providers. **Configuration and safety** - **FR-031**: Host configuration MUST make the active storage mode explicit when SQLite is selected. - **FR-032**: The existing no-configuration in-memory setup MUST continue to work. - **FR-033**: The feature MUST document that OpenTelemetry/exporter sinks are deferred and out of scope for this persistence slice. ### Key Entities *(include if feature involves data)* - **Structured Log Store**: Queryable storage for redacted structured log events and sources. - **Structured Log Live Feed**: Runtime stream of new events and dropped-event summaries for SignalR subscribers. - **Relational Structured Log Record**: Database representation of a `StructuredLogEvent` with indexed scalar columns and JSON payload columns. - **Relational Storage Dialect**: Provider-specific SQL and parameter behavior needed by the shared relational store. - **Structured Log Migration Runner**: FluentMigrator-based schema creation and upgrade service. - **Structured Log Retention Policy**: Settings and cleanup behavior that bound durable storage growth. ## Success Criteria *(mandatory)* ### Measurable Outcomes - **SC-001**: Existing structured logs unit and integration tests continue to pass with the in-memory default. - **SC-002**: New tests verify SQLite persistence survives service-provider or process recreation with the same database file. - **SC-003**: New tests verify SQLite recent queries honor level, category, source, workflow, correlation, trace/span, time range, and limit filters. - **SC-004**: New tests verify FluentMigrator creates the SQLite schema from an empty database. - **SC-005**: New tests verify retention cleanup removes expired or excess records only when retention settings are configured. - **SC-006**: New tests verify queued SQLite writes are flushed during graceful shutdown. - **SC-007**: New tests verify a full SQLite write queue drops newest events and reports dropped-write counts. - **SC-008**: New tests verify SQLite startup migrations run by default and can be disabled. - **SC-009**: New tests verify SQLite stores and filters UTC ISO-8601 timestamp text values consistently. - **SC-010**: New tests verify SQLite retention deletes no records by default. - **SC-011**: Documentation shows both zero-configuration in-memory setup and opt-in SQLite setup. - **SC-012**: The core structured logs module does not reference SQLite-specific types. ## Assumptions - The structured logs module from `004-diagnostics-structured-logs` is available and remains the owning diagnostics module. - SQLite is the only durable provider implemented in this feature. - Future relational providers can be added in separate specs and packages. - OpenTelemetry logs export, vendor sinks, raw console streaming, and trace/metric exploration remain out of scope. - The implementation can introduce new package dependencies such as FluentMigrator and Dapper where appropriate, with versions managed centrally.