842cf7c162
* Avoid null endpoint DTO metadata in tests * Enforce console logs hub read permission * Remove unused console logs hub import * Support mapped endpoint metadata in auth tests * Reduce console log capture throughput impact * Address Copilot console logs review * Refactor task scheduling to support tenant-level background work and enhance logging functionality. * Introduce ConsoleStreamHook for stdout/stderr tee and enhance logging validation. Adjust test cases and startup warnings for distributed lock provider usage. * Refactor console logging pipeline with capture optimization and new ConsoleLogsHost; update tests accordingly. * Add Ansi SGR parser for console logs and associated unit tests * Remove ANSI color renderings and parsers; integrate ConsoleLogScopeAccessor for improved logging context with workflow instance ID support. * Address console logs code quality feedback * Address PR review feedback * Preserve console logs extension points * Stabilize console logs host lifecycle * Address final automated review comments * Tighten console log capture shutdown * Address console log review feedback * Address follow-up review feedback * Cover final review feedback * Avoid recursive console provider initialization * Guard console host lease shutdown * Preserve console log scope and provider lifetime * Correlate console log scope fallback * Tighten console scope correlation * Expose host services during provider construction * Redact ANSI-normalized console lines * Add OpenTelemetry diagnostics backend foundation * Add OTLP HTTP ingestion parsing * Document OpenTelemetry diagnostics setup * Enforce OpenTelemetry hub permissions * Remove `ConsoleCaptureTee` and related services and tests * Add OpenTelemetry HTTP ingestion integration test * Use pipeline contributors for console log context * Update CShells package versions to 0.0.24-preview.132 * Add OpenTelemetry ingestion security tests * Add OpenTelemetry API authorization tests * Filter live console logs by workflow instance * Add OpenTelemetry hub tests * Add OpenTelemetry gRPC metadata hook * Assert OpenTelemetry workflow tags survive ingestion * Mark OpenTelemetry core build verified * Enhance console logging with activity execution metadata and extend test coverage. * Address console logs stream consumption comment * Wire OpenTelemetry diagnostics into core sample * Address Core diagnostics review feedback * Address Core Copilot follow-up feedback * Add OpenTelemetry metric instrument names * Address Core Copilot provider feedback * Address Core Copilot diagnostics follow-up * Address Core Copilot live feed feedback * Address Core Copilot store feedback * Integrate OpenTelemetry for logging, tracing, and metrics in ModularServer and update launch settings and docker-compose configuration. * Refactor to replace `ConsoleLogStream.Core` with `ConsoleLogStreaming.Core` across codebase and update `ConsoleStreamHook` installation. * Add diagnostics OpenTelemetry backend * Fix OpenTelemetry live hub subscription * Fix modular OpenTelemetry exporter endpoints * Add CShells logging configuration in appsettings.json * Remove obsolete unit tests and helper classes * Restore default activity exception handling * Simplify type serialization and alias management This commit refactors the internal type serialization and alias management system to reduce boilerplate, improve robustness, and simplify the developer experience: - Removed numerous explicit `ExpressionOptions` type alias registrations across various modules. - Updated `TypeJsonConverter` and polymorphic serialization to reliably handle types using assembly-qualified names when a short alias is not explicitly registered. - Streamlined `ExcludeFromHashConverter` to strictly adhere to `ExcludeFromHashAttribute` for hash calculations, removing complex `JsonIgnoreCondition` logic. - Eliminated several helper classes (`WorkflowJsonTypeResolver`, `WorkflowTypeValidator`, `IWorkflowTypeRegistry`, `WorkflowFactoryDictionary`, `JavaScriptExceptionTypeAliasRegistrar`, `WorkflowRuntimeTypeAliasRegistrar`) and their associated unit tests, simplifying the codebase. Additionally, this commit introduces a comprehensive markdown document (`product-website-feature-source.md`) outlining Elsa's core features, Studio capabilities, extension ecosystem, and architectural selling points, intended as source material for the product website. * Refine type serialization for improved robustness and alias handling This commit further enhances the type serialization and deserialization mechanisms: * Centralizes type resolution and alias management through `IWellKnownTypeRegistry` and `WorkflowJsonTypeResolver`. * Prioritizes registered type aliases when serializing type metadata in `PolymorphicObjectConverter`, resulting in more concise JSON output. * Enhances deserialization in `PolymorphicObjectConverter` and `VariableMapper` to gracefully handle unknown or non-instantiable types, providing fallbacks and logging warnings. * Simplifies `TypeJsonConverter` by delegating complex type resolution logic to the `WorkflowJsonTypeResolver`. * Adds `JsonArray` to the well-known type aliases for direct recognition. * Fix console logs packaging and workflow type resolution * Fix console log metadata and type resolution * Address Copilot review feedback * Enhance type resolution, improve console log handling, and update tests - Streamlined `WorkflowDictionaryExtensions` for better workflow registration validation. - Refined `ConsoleLogsAuthorizationTests` with the new `SetJsonRequest` helper to improve test requests handling. - Updated `OrderDefinition` to ignore JSON serialization for `KeySelector`. - Enhanced `WorkflowRuntimeFeature` for improved workflow registration and type alias configuration. - Added tests to ensure `ConsoleLogProvider` metadata filtration in various scenarios. - Improved type serialization logic in `WorkflowJsonTypeResolver`. - Updated README to fix references related to diagnostics. - Optimized `ExcludeFromHashConverter` for property serialization conditions. - Modified `TriggerIndexer` for streamlined trigger management. - Tested payload checks in `PublishEventTests`. - Adjusted `Endpoint` in `ConsoleLogs` for automatic JSON request handling. - Ensured registration of workflow type aliases in `WorkflowsFeature`. * Restore CLR workflow registration compatibility * Align JSON island serialization fixtures * Add Console Logs Services and Enhance Endpoint Handling - Introduced `ActivityExecutionsEndpointTests` to validate route exposure. - Added `ConsoleLogCaptureHostedService` for console log streaming. - Implemented `ConsoleStreamJsonConverter` for JSON conversion of console streams. - Developed `ElsaConsoleLogRecentBuffer` to handle recent log buffering. - Updated `ConsoleLogsAuthorizationTests` with new test cases for stream filter mapping. - Consolidated console log provider dependencies and registration, including recent buffering. - Enhanced `ElsaConsoleLogProvider` to use recent buffer for filtering. - Adjusted `Program.cs` for streamlined logging service setup. * Enhance type resolution and test coverage; streamline console log integration - Added `ConsoleStreamHook` for streamlined log streaming. - Updated `WorkflowJsonTypeResolverTests` to improve type resolution and test new scenarios. - Simplified type resolution by removing trusted assembly checks. * Fix CI smoke and package restore failures * Fix Docker smoke image project paths * Fix Docker Python runtime packages * Fix Docker CA smoke teardown * Refresh Elsa roadmap * Implement background processors and mediation coordination - Added `BackgroundCommandProcessor`, `BackgroundJobProcessor`, and `BackgroundNotificationProcessor` classes for handling commands, jobs, and notifications, respectively. - Introduced `MediatorBackgroundProcessingCoordinator` to coordinate the execution of all background processors. - Implemented `MediatorBackgroundTask` for wrapping `MediatorBackgroundProcessingCoordinator` in `BackgroundTask`. - Added unit tests for `MediatorBackgroundTask` to ensure proper start and stop behavior. - Refactored `BackgroundCommandSenderHostedService` to utilize `BackgroundCommandProcessor`. - Introduced 'elsa-roadmap-refresh' skill configuration for roadmap updates. * Address workflow type resolution review feedback * Address follow-up review feedback * Restore recent console logs execute path * Address Copilot follow-up review * Decouple workflow JSON aliases from expressions * Fix workflow management unit test setup * Fix console logs recent endpoint handler shape * Respect workflow JSON strict type aliases * Remove unused console log contracts reference * Address Copilot review feedback * Address Copilot follow-up comments * Synchronize ring buffer dropped count * Address background processor strategy replay
105 lines
4.2 KiB
Markdown
105 lines
4.2 KiB
Markdown
# Data Model: Diagnostics OpenTelemetry
|
|
|
|
## Telemetry Resource
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `ResourceKey` | Stable key derived from service name, instance ID, and resource attributes. |
|
|
| `ServiceName` | From `service.name` or `OTEL_SERVICE_NAME`; fallback is `unknown_service`. |
|
|
| `ServiceInstanceId` | From `service.instance.id` when present. |
|
|
| `ServiceVersion` | From `service.version` when present. |
|
|
| `DeploymentEnvironment` | From `deployment.environment` when present. |
|
|
| `Attributes` | Redacted resource attributes. |
|
|
| `FirstSeen`, `LastSeen` | Backend receive timestamps. |
|
|
| `Status` | Active, stale, or disconnected. |
|
|
|
|
## Telemetry Trace
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `TraceId` | Required trace identifier. |
|
|
| `RootSpanId` | Root span when known. |
|
|
| `Name` | Root span name or synthesized trace label. |
|
|
| `ResourceKeys` | Participating resources. |
|
|
| `StartTime`, `EndTime`, `Duration` | Derived from spans. |
|
|
| `Status` | Ok, error, unset, or mixed. |
|
|
| `SpanCount`, `ErrorCount` | Derived counters. |
|
|
| `WorkflowInstanceId`, `WorkflowDefinitionId` | Elsa span attributes when present. |
|
|
| `LastReceivedAt` | Backend receive timestamp. |
|
|
|
|
## Telemetry Span
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `TraceId`, `SpanId`, `ParentSpanId` | Trace hierarchy. |
|
|
| `ResourceKey` | Emitting resource. |
|
|
| `Name`, `Kind` | OTEL span metadata. |
|
|
| `StartTime`, `EndTime`, `Duration` | Timing. |
|
|
| `StatusCode`, `StatusDescription` | OTEL status. |
|
|
| `Attributes` | Redacted span attributes. |
|
|
| `Events` | Redacted events. |
|
|
| `Links` | Span links. |
|
|
| `Workflow*`, `Activity*`, `TenantId`, `CorrelationId` | Existing `Elsa.Workflows` semantic attributes when present: `workflow.*`, `workflow.activity.*`, and `elsa.tenant.id`. |
|
|
| `ReceivedAt` | Backend receive timestamp. |
|
|
|
|
## Metric Instrument and Point
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `Name`, `Description`, `Unit`, `Type` | Instrument metadata. |
|
|
| `ResourceKey` | Emitting resource. |
|
|
| `Series` | Bounded collection by attribute set. |
|
|
| `Timestamp`, `StartTimestamp` | Point timestamps. |
|
|
| `Attributes` | Redacted point attributes. |
|
|
| `Value` | Sum/gauge value. |
|
|
| `Count`, `Sum`, `Min`, `Max`, `Buckets` | Histogram fields when present. |
|
|
| `Temporality` | Cumulative or delta. |
|
|
| `DroppedPointCount` | Capacity drop count. |
|
|
|
|
## OTLP Log Record
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `Id` | Backend assigned ID. |
|
|
| `Timestamp`, `ObservedTimestamp`, `ReceivedAt` | OTLP and backend timing. |
|
|
| `ResourceKey` | Emitting resource. |
|
|
| `SeverityText`, `SeverityNumber` | OTEL severity. |
|
|
| `Body` | Redacted body. |
|
|
| `TraceId`, `SpanId` | Correlation fields. |
|
|
| `Attributes` | Redacted log attributes. |
|
|
|
|
## Collector Configuration
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `HttpEndpoint` | Base endpoint suitable for `OTEL_EXPORTER_OTLP_ENDPOINT` with HTTP/protobuf. |
|
|
| `GrpcEndpoint` | Endpoint suitable for gRPC when enabled; null when unavailable. |
|
|
| `GrpcEnabled` | Whether gRPC ingestion is enabled. |
|
|
| `GrpcDisabledReason` | Optional non-secret explanation when disabled. |
|
|
| `RequiredHeaders` | Required ingestion header names without secret values. |
|
|
| `RecommendedEnvironment` | Copyable non-secret OTEL environment variable values. |
|
|
| `IsLoopbackOnly` | Whether ingestion is restricted to loopback. |
|
|
| `RequiresApiKey` | Whether configured ingestion protection is required for advertised endpoints. |
|
|
|
|
## Storage Diagnostics
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `DroppedSpanCount` | Spans dropped by store capacity. |
|
|
| `DroppedMetricPointCount` | Metric points dropped by capacity. |
|
|
| `DroppedLogRecordCount` | OTLP logs dropped by capacity. |
|
|
| `DroppedLiveUpdateCount` | Live updates dropped by subscriber capacity. |
|
|
| `ResourceCount`, `TraceCount`, `MetricSeriesCount` | Current counts. |
|
|
|
|
## Storage Capacity Policy
|
|
|
|
| Field | Notes |
|
|
|-------|-------|
|
|
| `MaxResources` | Default at least 500. |
|
|
| `MaxTraces` | Default at least 2,000. |
|
|
| `MaxSpans` | Default at least 10,000. |
|
|
| `MaxMetricPoints` | Default at least 20,000 across all metric series. |
|
|
| `MaxLogRecords` | Default at least 10,000. |
|
|
| `MaxLiveUpdatesPerSubscriber` | Default at least 1,000 queued updates per SignalR subscriber. |
|
|
| `OverflowPolicy` | Drop oldest item in the signal-specific buffer and increment the matching dropped counter. |
|