Sipke Schoorstra b255e0b4c3

[codex] Expose structured log storage diagnostics (#7446 )

* Expose structured log storage diagnostics

* Address structured log diagnostics review

* Fix checked storage diagnostics aggregation

2026-05-13 16:32:23 +02:00

3.6 KiB

Raw Blame History

Quickstart: Structured Log Persistence

Default in-memory storage

No durable storage configuration is required:

services.AddElsa(elsa =>
{
    elsa.UseStructuredLogs(options =>
    {
        options.RecentLogCapacity = 5_000;
        options.MaxRecentLogQuerySize = 1_000;
    });
});

This keeps the current bounded in-memory recent history and live SignalR streaming behavior.

SQLite durable storage

Add the SQLite persistence package and configure a database file:

services.AddElsa(elsa =>
{
    elsa.UseStructuredLogs(structuredLogs =>
    {
        structuredLogs.UseSqliteStorage("Data Source=elsa-structured-logs.db", sqlite =>
        {
            sqlite.RunMigrationsOnStartup = true;
            sqlite.Relational.WriteQueue.Capacity = 10_000;
            sqlite.Relational.WriteQueue.BatchSize = 100;
        });
    });
});

Then map the existing structured logs endpoints and hub:

app.UseStructuredLogs();

Studio continues to use:

/diagnostics/structured-logs/recent
/diagnostics/structured-logs/sources
/diagnostics/structured-logs/storage
/elsa/hubs/diagnostics/structured-logs

Migrations

SQLite storage runs FluentMigrator migrations when RunMigrationsOnStartup is enabled. For SQLite this is the recommended default.

For future shared relational providers such as SQL Server or PostgreSQL, production deployments may choose to run migrations once during deployment instead of from every application instance. Multi-instance startup locking must be documented by each provider.

Retention

SQLite storage does not delete persisted log entries by default. Configure Retention.MaxAge, Retention.MaxRows, or both to bound durable storage growth:

sqlite.Relational.Retention.MaxAge = TimeSpan.FromDays(14);
sqlite.Relational.Retention.MaxRows = 250_000;
sqlite.Relational.Retention.CleanupOnStartup = true;

Write buffering

SQLite writes are batched through a bounded background queue. Graceful shutdown flushes queued events where possible. If the process crashes, queued-but-unflushed events can be lost. If the queue is full, newly received events are dropped and dropped-write counts are reported.

Validation

Run targeted checks after implementation:

dotnet build src/modules/Elsa.Diagnostics.StructuredLogs/Elsa.Diagnostics.StructuredLogs.csproj
dotnet build src/modules/Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite/Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite.csproj
dotnet test test/unit/Elsa.Diagnostics.StructuredLogs.UnitTests/Elsa.Diagnostics.StructuredLogs.UnitTests.csproj
dotnet test test/integration/Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite.IntegrationTests/Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite.IntegrationTests.csproj

Manual validation:

Start Elsa Server with SQLite structured log storage enabled.
Emit several ILogger records with different levels, categories, workflow IDs, and correlation IDs.
Query recent logs from Studio and verify filters work.
Restart the host using the same SQLite database file.
Query recent logs again and verify events from before the restart are still available.
Verify timestamps are stored and filtered as UTC ISO-8601 values.
Lower retention settings in a test environment and verify cleanup removes old or excess rows.
Saturate the write queue in a test environment and verify newest events are dropped with visible dropped-write counts.

Out of scope

This feature does not add OTLP, Logstash, Datadog, Splunk, Loki, Seq, or Parquet exporters. Those can be added later as sinks/exporters after persistence is stable.

3.6 KiB Raw Blame History