elsa-core/specs/005-structured-log-persistence/plan.md

Implementation Plan: Structured Log Persistence

Branch: 005-structured-log-persistence | Date: 2026-05-13 | Spec: spec.md Input: Feature specification from /specs/005-structured-log-persistence/spec.md

Summary

Refactor diagnostics structured logs so queryable storage is replaceable while the existing IStructuredLogProvider remains the REST/SignalR facade. Keep bounded in-memory storage as the default and add opt-in SQLite durable storage backed by shared relational persistence. Use FluentMigrator for schema management, explicit SQL/Dapper-style runtime operations, async batched writes, a bounded drop-newest write queue, startup migrations enabled by default, UTC ISO-8601 timestamp text, and opt-in retention.

Technical Context

Language/Version: C# latest, nullable reference types enabled, implicit usings enabled. Primary Dependencies: Existing Elsa.Diagnostics.StructuredLogs, Microsoft.Extensions.Logging, Microsoft.Extensions.Options, Microsoft.AspNetCore.SignalR, Elsa feature/module infrastructure, FastEndpoints through Elsa API endpoint patterns, FluentMigrator runner packages, SQLite ADO.NET provider, and optionally Dapper for relational operations. Storage: Bounded in-memory store by default; opt-in SQLite durable store through shared relational persistence. SQLite stores Timestamp and ReceivedAt as UTC ISO-8601 text and stores exception, scope, and property payloads as JSON text. Testing: xUnit unit and integration tests under existing structured logs projects plus new relational unit tests and SQLite integration tests. Target Platform: ASP.NET Core Elsa Server on the repository's supported .NET target frameworks. Project Type: Modular .NET libraries inside the existing Elsa solution. Performance Goals: Preserve non-blocking ILogger capture. SQLite writes use async batching and a bounded queue; full queues drop newly received events and report dropped-write counts. Constraints: No EF Core persistence for this feature. OpenTelemetry, vendor exporters, raw console capture, and trace/metric exploration remain out of scope. Redaction must run before persistence. SQLite may lose queued-but-unflushed events after an ungraceful process crash. Scale/Scope: Core structured logs storage refactor, shared relational package, SQLite provider package, tests, docs, and Speckit artifacts.

Constitution Check

GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.

Principle	Verdict	Evidence
I. Modular Architecture	PASS	Core contracts stay in Elsa.Diagnostics.StructuredLogs; relational and SQLite persistence live in focused provider packages under src/modules/.
II. Composition & Extensibility	PASS	Store, live feed, dialect, migration runner, write queue, and retention services are replaceable.
III. Convention-Driven Design	PASS	Public APIs continue the StructuredLog/StructuredLogs naming established by feature 004 and use American English.
IV. Async & Pipeline Execution	PASS	Append, query, migration, cleanup, flush, and shutdown paths are async/cancellation-aware.
V. Testing Discipline	PASS	Plan includes unit/integration coverage for in-memory compatibility, SQLite durability, migrations, queue overflow, timestamp storage, and retention.
VI. Trunk-Based Development	PASS	SQLite persistence can ship independently from future OTLP or vendor sink work.
VII. Simplicity, SRP, DRY & KISS	PASS	EF Core and broad vendor exporter support are intentionally excluded; abstractions serve documented persistence extensibility.

Project Structure

Documentation (this feature)

specs/005-structured-log-persistence/
├── spec.md
├── plan.md
├── research.md
├── data-model.md
├── quickstart.md
├── contracts/
│   └── persistence-contract.md
├── checklists/
│   └── requirements.md
└── tasks.md

Source Code (repository root)

src/modules/
├── Elsa.Diagnostics.StructuredLogs/
│   ├── Contracts/
│   │   ├── IStructuredLogProvider.cs
│   │   ├── IStructuredLogStore.cs
│   │   ├── IStructuredLogLiveFeed.cs
│   │   └── IStructuredLogSink.cs
│   ├── Providers/InMemory/
│   └── Services/DefaultStructuredLogProvider.cs
├── Elsa.Diagnostics.StructuredLogs.Persistence.Relational/
│   ├── Contracts/
│   ├── Features/
│   ├── Migrations/
│   ├── Models/
│   ├── Options/
│   ├── Services/
│   ├── ShellFeatures/
│   └── Stores/
└── Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite/
    ├── Extensions/
    ├── Features/
    ├── Options/
    ├── Services/
    └── ShellFeatures/

test/unit/
├── Elsa.Diagnostics.StructuredLogs.UnitTests/
└── Elsa.Diagnostics.StructuredLogs.Persistence.Relational.UnitTests/

test/integration/
└── Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite.IntegrationTests/

Structure Decision: Keep runtime capture, API, hub, and Studio-facing contracts in the existing structured logs module. Add relational and SQLite packages only for durable persistence. Do not create OTLP or vendor sink packages in this feature.

Phase 0 Output

See research.md.

Resolved decisions:

• Keep in-memory storage as the default.
• Split storage/live-feed responsibilities from the existing provider facade.
• Add SQLite as the first durable provider.
• Build SQLite on a shared relational persistence package.
• Use FluentMigrator for schema management and run SQLite migrations on startup by default with opt-out.
• Use explicit SQL/Dapper-style operations for appending, querying, and retention.
• Use async batched SQLite writes with graceful-shutdown flush.
• Use a bounded write queue that drops newest events when full and reports dropped-write counts.
• Store SQLite timestamps as UTC ISO-8601 text.
• Make retention opt-in; delete no records unless max age or max rows is configured.
• Defer OpenTelemetry and vendor sink/exporter packages.

Phase 1 Output

• data-model.md
• contracts/persistence-contract.md
• quickstart.md

Post-Design Constitution Re-Check

Principle	Verdict	Post-design evidence
I. Modular Architecture	PASS	Persistence packages are separate from the core diagnostics structured logs module.
II. Composition & Extensibility	PASS	Future relational providers can reuse the relational store and provide provider-specific services.
III. Convention-Driven Design	PASS	Configuration and package names follow Elsa module conventions and American English.
IV. Async & Pipeline Execution	PASS	Contracts and planned services use async append/query/migration/cleanup/flush methods.
V. Testing Discipline	PASS	Story tasks include independent tests for each deliverable slice and clarified edge behavior.
VI. Trunk-Based Development	PASS	The SQLite slice is independently releasable.
VII. Simplicity, SRP, DRY & KISS	PASS	The design avoids EF Core and skips exporter scope while keeping relational extensibility focused.

Phase 2 Handoff

Use tasks.md as the implementation backlog. Suggested order:

1. Introduce storage/live-feed contracts while preserving current behavior.
2. Convert the in-memory provider into reusable in-memory store/live-feed pieces behind the existing facade.
3. Add shared relational persistence contracts, record model, dialect, migration runner, retention service, write queue, and store implementation.
4. Add SQLite provider registration, dialect, migration runner configuration, startup migration option, and tests.
5. Update README/quickstart and run targeted builds/tests.

Complexity Tracking

No constitution violations identified.