elsa-core/doc/wiki/diagnostics-structured-logs.md

Diagnostics Structured Logs

Elsa.Diagnostics.StructuredLogs captures semantic ILogger events from an Elsa host, redacts sensitive data, keeps a recent queryable buffer, exposes REST endpoints, and streams live events to Studio over SignalR.

Start in src/modules/Elsa.Diagnostics.StructuredLogs.

Scope

This module captures structured ILogger records only. It does not capture raw stdout/stderr console streams, traces, metrics, or OpenTelemetry spans. Raw console output is covered by the sibling Elsa.Diagnostics.ConsoleLogs module — see Diagnostics Console Logs.

Feature Wiring

StructuredLogsFeature:

• registers FastEndpoints assembly
• calls AddStructuredLogsServices
• adds FastEndpoints from the module

AddStructuredLogsServices registers:

• SignalR
• StructuredLogsOptions
• source registry
• redactor
• in-memory store
• in-memory live feed
• default provider facade
• subscription manager
• StructuredLogLoggerProvider as an ILoggerProvider

Core Contracts

Contract	Purpose
IStructuredLogProvider	REST/SignalR facade used by endpoints and clients.
IStructuredLogStore	Queryable storage abstraction.
IStructuredLogLiveFeed	Live event publication/subscription abstraction.
IStructuredLogSink	Event ingestion boundary.
IStructuredLogRedactor	Redacts properties and text before storage/live delivery.
IStructuredLogSourceRegistry	Tracks source metadata and health.
IStructuredLogStorageDiagnostics	Provider-neutral diagnostics such as dropped durable writes.

Event Flow

sequenceDiagram
    participant App as App ILogger
    participant Provider as StructuredLogLoggerProvider
    participant Redactor as IStructuredLogRedactor
    participant Store as IStructuredLogStore
    participant Feed as IStructuredLogLiveFeed
    participant Hub as StructuredLogsHub
    participant Studio as Studio

    App->>Provider: Log event + scopes
    Provider->>Redactor: redact event
    Redactor->>Store: append event
    Redactor->>Feed: publish event
    Feed->>Hub: live event
    Hub->>Studio: SignalR stream
    Studio->>Store: recent query through REST

In-Memory Provider

The default provider keeps recent logs in process:

• InMemoryStructuredLogStore
• InMemoryStructuredLogLiveFeed
• RingBuffer

This is bounded and process-local. In clustered deployments, each node has its own source identity and in-memory history unless durable/shared persistence is configured.

REST And SignalR Surface

REST endpoints:

• GET|POST /elsa/api/diagnostics/structured-logs/recent
• GET /elsa/api/diagnostics/structured-logs/sources
• GET /elsa/api/diagnostics/structured-logs/storage

Endpoint code is under Endpoints/StructuredLogs.

SignalR:

• Hub: StructuredLogsHub
• Client contract: IStructuredLogsClient
• Mapping: MapStructuredLogsHub
• App extension: UseStructuredLogs

The README states the hub is mapped at /elsa/hubs/diagnostics/structured-logs.

Authorization

The endpoints require read:diagnostics:structured-logs, defined in StructuredLogsPermissions. The SignalR hub requires an authenticated user.

Redaction

Events pass through IStructuredLogRedactor before buffering or streaming. Configuration lives in StructuredLogsOptions. Extend sensitive property names and text patterns there.

SQLite Persistence

Durable SQLite storage is available through the relational and SQLite packages:

• design plan: specs/005-structured-log-persistence/plan.md
• quickstart: specs/005-structured-log-persistence/quickstart.md
• relational package: Elsa.Diagnostics.StructuredLogs.Persistence.Relational
• SQLite package: Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite

Configuration example from the SQLite README:

services.AddElsa(elsa =>
{
    elsa.UseStructuredLogs(structuredLogs =>
    {
        structuredLogs.UseSqliteStorage("Data Source=elsa-structured-logs.db", sqlite =>
        {
            sqlite.RunMigrationsOnStartup = true;
            sqlite.Relational.WriteQueue.Capacity = 10_000;
            sqlite.Relational.WriteQueue.BatchSize = 100;
        });
    });
});

Relational Persistence Design

AddRelationalStructuredLogPersistence registers:

• RelationalStructuredLogMapper
• RelationalStructuredLogSqlBuilder
• RelationalStructuredLogStore
• StructuredLogWriteBuffer
• StructuredLogRetentionService
• IStructuredLogStore as the write buffer
• IStructuredLogWriteBuffer
• IStructuredLogStorageDiagnostics
• hosted service for the write buffer

The write buffer uses a bounded queue. If the queue is full, newest events are dropped and the dropped-write count is reported through storage diagnostics.

SQLite Provider Boundary

AddSqliteStructuredLogPersistence supplies provider-specific services:

• IRelationalStructuredLogConnectionFactory
• IRelationalStructuredLogDialect
• IStructuredLogSchemaMigrator
• startup migration/cleanup hosted service

The core structured logs package must remain unaware of SQLite. Future relational providers should copy this boundary: provider package supplies connection factory, dialect, migrator, and option binding; relational package supplies shared store behavior.

Tests

Relevant tests:

• test/unit/Elsa.Diagnostics.StructuredLogs.UnitTests
• test/integration/Elsa.Diagnostics.StructuredLogs.IntegrationTests
• test/unit/Elsa.Diagnostics.StructuredLogs.Persistence.Relational.UnitTests
• test/integration/Elsa.Diagnostics.StructuredLogs.Persistence.Sqlite.IntegrationTests

Run targeted structured-log tests before broader builds when touching this area.