elsa-core/specs/006-diagnostics-console-logs/spec.md

# Feature Specification: Diagnostics Console Logs

**Feature Branch**: `006-diagnostics-console-logs`  
**Created**: 2026-05-18  
**Status**: Draft  
**Input**: User description: "Create a Core feature spec for diagnostics console streaming based on the existing roadmap at specs/diagnostics-console-streaming-roadmap.md and the surrounding context from specs/003-live-server-logs, specs/004-diagnostics-structured-logs, and specs/005-structured-log-persistence. Include requirements for backend capture, buffering, endpoints, SignalR hub, permissions, source identity, redaction, and provider boundaries. Avoid implementation code changes."

## Clarifications

### Session 2026-05-18

- Q: How should Core expose partial stdout/stderr writes that have not ended with a newline? -> A: Buffer until complete.
- Q: How should Core handle console lines longer than the configured maximum line length? -> A: Truncate oversized lines.
- Q: What should Core do with ANSI escape sequences by default? -> A: Strip ANSI by default.
- Q: What permission boundary should Core use for source listings? -> A: Same dedicated permission.
- Q: What content should console log providers receive and retain? -> A: Redacted content only.

## User Scenarios & Testing *(mandatory)*

### User Story 1 - Tail raw backend console output (Priority: P1)

An administrator or developer enables the Core console logs feature and watches recent plus live stdout and stderr lines from the backend process without using shell access.

**Why this priority**: Raw process console output is the core capability and is intentionally separate from structured log records.

**Independent Test**: Enable the console logs feature, write distinct lines to stdout and stderr, request recent lines, subscribe to the live stream, and verify callers receive ordered, source-aware console line events while the original console output still reaches its normal destination.

**Acceptance Scenarios**:

1. **Given** diagnostics console logs are enabled and the caller is authorized, **When** the backend writes complete lines to stdout and stderr, **Then** the caller receives console line events with stream identity, text, timestamps, sequence, and source identity.
2. **Given** console capture is enabled, **When** the backend writes to stdout or stderr, **Then** existing console behavior is preserved and output remains visible to the host environment.
3. **Given** the caller requests recent console lines before subscribing live, **When** recent matching lines exist, **Then** the server returns a bounded ordered backfill before the caller receives new live lines.

---

### User Story 2 - Filter, secure, and redact console output (Priority: P2)

An operator narrows noisy console output by source, stream, text, and time while the backend enforces a dedicated diagnostics permission and redacts sensitive data before data leaves the backend.

**Why this priority**: Console output can contain secrets and high-volume operational noise. The feature is not safe for real hosts without authorization, filtering, and redaction.

**Independent Test**: Connect authorized and unauthorized callers, write console lines containing secret-like values, apply filters, and verify unauthorized access is rejected while authorized callers only receive redacted lines matching their filters.

**Acceptance Scenarios**:

1. **Given** an unauthenticated or unauthorized caller, **When** the caller requests recent console lines, source listings, or a live subscription, **Then** access is rejected.
2. **Given** redaction rules are configured, **When** stdout or stderr contains matching sensitive values, **Then** recent and live events replace those values with a redaction marker before exposure.
3. **Given** the caller filters by source, stream, text, or time, **When** matching and non-matching console lines are available, **Then** only matching redacted lines are returned or streamed.

---

### User Story 3 - Identify console sources in clustered deployments (Priority: P3)

An operator diagnosing a clustered Elsa deployment can view a merged stream across known backend sources and filter to one source such as a process, pod, or container.

**Why this priority**: Source identity must be part of the first contract so the console viewer does not become single-process-only and can later use shared providers without changing Studio-facing behavior.

**Independent Test**: Simulate multiple console log sources through the provider boundary, request sources, subscribe to a merged stream, and then filter to one source while preserving source health and dropped-line metadata.

**Acceptance Scenarios**:

1. **Given** multiple sources publish console lines through the configured provider, **When** the caller subscribes without a source filter, **Then** the server streams a merged ordered view with each line's source identity.
2. **Given** the caller filters by a source ID, **When** lines arrive from multiple sources, **Then** only lines from the selected source are returned or streamed.
3. **Given** a source stops sending heartbeats or lines, **When** the caller lists sources, **Then** the source is marked stale or disconnected without immediately losing its recent history.

### Edge Cases

- Console writes arrive as partial lines or without a trailing newline.
- A console line exceeds the configured maximum line length.
- Output arrives faster than the recent buffer or subscriber queues can handle.
- A subscriber disconnects while console lines continue to arrive.
- A caller changes filters while subscribed.
- Console output contains ANSI escape sequences.
- Console output contains secrets in raw text or source metadata.
- Multiple sources have clock skew or overlapping sequence values.
- The console logs feature writes its own diagnostics messages and risks feeding them back into the captured console stream.
- The configured provider is unavailable at startup or becomes unavailable during streaming.

## Requirements *(mandatory)*

### Functional Requirements

**Module boundary and feature identity**

- **FR-001**: The backend MUST provide an opt-in Core diagnostics module named `Elsa.Diagnostics.ConsoleLogs` or an equivalent diagnostics console logs name confirmed before implementation.
- **FR-002**: The console logs module MUST be separate from `Elsa.Diagnostics.StructuredLogs` and MUST NOT replace, rename, or parse structured log records.
- **FR-003**: The module MUST advertise a diagnostics console logs remote feature so Studio can detect whether the backend supports console streaming.
- **FR-004**: The feature MUST document that raw stdout/stderr console streaming is distinct from structured logs, structured log persistence, trace waterfalls, metrics, and OpenTelemetry exploration.

**Backend capture and line behavior**

- **FR-005**: The feature MUST capture writes to stdout and stderr as raw console output while preserving the host's existing console output behavior.
- **FR-006**: Captured output MUST be emitted as line-oriented console events with stream identity of `stdout` or `stderr`.
- **FR-007**: Captured console events MUST include ID, timestamp, received timestamp, source-local sequence, stream, text, source identity, truncation indicator, and dropped-line metadata when available.
- **FR-008**: Partial writes MUST be buffered and MUST NOT be exposed as separate fragment events; Core completes and emits a line only on newline, maximum line length, or configurable idle flush timeout.
- **FR-009**: Lines longer than the configured maximum line length MUST be truncated to the configured maximum, emitted as a single console line event, and marked with a truncation indicator.
- **FR-010**: ANSI escape sequences MUST be stripped by default before exposure, and hosts MUST be able to configure preservation of terminal formatting when needed.
- **FR-011**: The feature MUST avoid recursively capturing console diagnostics generated by the console logs feature itself unless explicitly enabled for troubleshooting.

**Buffering and backpressure**

- **FR-012**: The feature MUST keep a bounded recent-history buffer for initial backfill and MUST never grow memory without bound.
- **FR-013**: The feature MUST use bounded live subscriber behavior and MUST track dropped-line counts when buffers or subscriber queues overflow.
- **FR-014**: Recent-line requests MUST enforce a server-side maximum result count regardless of caller input.
- **FR-015**: Dropped-line summaries MUST identify the affected source and reason when that information is known.

**Endpoints and live transport**

- **FR-016**: The backend MUST expose an authenticated recent console lines endpoint for initial backfill, for example `/diagnostics/console-logs/recent`.
- **FR-017**: The backend MUST expose an authenticated console sources endpoint, for example `/diagnostics/console-logs/sources`.
- **FR-018**: The backend MUST expose an authenticated live console logs SignalR hub, for example `/elsa/hubs/diagnostics/console-logs`.
- **FR-019**: Recent queries and live subscriptions MUST support filters for source ID, stream, free-text query, time range, and maximum result count.
- **FR-020**: The live hub MUST allow a caller to subscribe, update filters without reconnecting, and unsubscribe.
- **FR-021**: The live hub MUST send console line events, dropped-line summaries, and source status changes to subscribed callers.

**Authorization and redaction**

- **FR-022**: Recent-line endpoints, source-listing endpoints, and live console log hubs MUST all require the same dedicated permission such as `read:diagnostics:console-logs`.
- **FR-023**: The feature MUST use Elsa's existing authentication, authorization, and cross-origin access patterns.
- **FR-024**: Redaction MUST run before console lines or source metadata are stored in recent buffers, streamed live, or returned by endpoints.
- **FR-025**: Default redaction MUST mask common secret indicators such as authorization, bearer tokens, passwords, secrets, API keys, cookies, connection strings, and similarly named values.
- **FR-026**: Hosts MUST be able to configure additional redaction rules and replacement text.
- **FR-027**: Redaction MUST apply to raw line text and to sensitive source metadata fields before data leaves the backend.
- **FR-027a**: Console log providers MUST receive, store, stream, and list only redacted console line text and redacted source metadata; raw unredacted console content MUST remain inside the capture/redaction boundary.

**Source identity and provider boundaries**

- **FR-028**: Every console line MUST include a source descriptor with source ID, display name, service name, process ID, machine name, and, when available, pod name, container name, namespace, and node name.
- **FR-029**: Sources MUST expose last-seen time and health such as connected, stale, or disconnected.
- **FR-030**: The default provider MUST support in-process capture for local development, tests, and single-node hosts.
- **FR-031**: The provider boundary MUST allow future shared or external providers to aggregate console lines across multiple Core instances without changing Studio-facing contracts.
- **FR-032**: Direct Kubernetes, Docker, orchestrator log API, vendor sink, and OpenTelemetry integrations MUST remain out of scope for this feature unless added by later specs.
- **FR-033**: Merged streams MUST provide deterministic ordering when source timestamps overlap or clocks differ, using received order or another stable tiebreaker.

**Configuration and operations**

- **FR-034**: The feature MUST provide host options for recent buffer capacity, subscriber/channel capacity, maximum recent query size, maximum line length, idle flush timeout, ANSI handling, source heartbeat timeout, redaction rules, and provider selection.
- **FR-035**: Provider failures MUST be surfaced to recent queries or live subscriptions with safe error information and without exposing unredacted console content.
- **FR-036**: The feature MUST include validation coverage for capture, stream identity, filtering, buffering, dropped counts, authorization, redaction, source health, provider boundaries, and feedback-loop prevention.

### Key Entities *(include if feature involves data)*

- **Console Log Line**: A redacted raw stdout or stderr line with sequence, timestamps, stream identity, text, truncation state, source identity, and dropped-line context.
- **Console Log Source**: The process, pod, container, machine, or external provider source that produced console output.
- **Console Log Filter**: Criteria used for recent queries and live subscriptions, including source, stream, text, time range, and take limit.
- **Console Log Provider**: Replaceable provider that stores recent console lines, streams future lines, lists sources, and reports dropped-line/source status.
- **Console Capture Tee**: Capture boundary that observes stdout/stderr while preserving the original console destination.
- **Console Redaction Rule**: Configured matching rule that masks sensitive raw line text or source metadata before exposure.
- **Console Log Subscription**: Live connection with active filters, bounded queue behavior, and dropped-line summaries.

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-001**: In single-process validation, authorized subscribers receive 100% of complete stdout and stderr test lines up to configured subscriber capacity within 1 second of capture.
- **SC-002**: Recent-line queries never return more than the configured maximum number of lines.
- **SC-003**: Unauthorized callers are rejected for 100% of recent-line, source-list, and live hub access attempts.
- **SC-004**: Default redaction masks common secret patterns in raw console line text and sensitive source metadata in validation tests.
- **SC-005**: Under sustained overload, memory remains bounded and dropped-line counts are visible through recent results or live summaries.
- **SC-006**: In simulated multi-source validation, merged results include lines from all active sources and source filtering returns only the selected source.
- **SC-007**: A source that stops sending lines or heartbeats is marked stale within the configured heartbeat timeout.
- **SC-008**: Console writes continue reaching the host's original stdout/stderr destination while capture is enabled.
- **SC-009**: Documentation or feature metadata clearly separates console logs from structured logs, structured log persistence, and future OpenTelemetry diagnostics.

## Assumptions

- Studio will implement a paired feature spec in the `elsa-studio` repository and is owned by another worker.
- The Core route names, hub route, remote feature name, and permission use diagnostics console logs naming unless later renamed consistently before implementation.
- In-process capture is the first slice; cluster-ready behavior is provided through source identity and provider boundaries rather than direct orchestrator APIs.
- Recent console history is operational troubleshooting data, not durable audit logging.
- Redacted console lines are the only form exposed by providers, endpoints, hubs, and buffers.
- Hosts that need long-term log retention should continue using their existing observability or platform log systems.