* Avoid null endpoint DTO metadata in tests * Enforce console logs hub read permission * Remove unused console logs hub import * Support mapped endpoint metadata in auth tests * Reduce console log capture throughput impact * Address Copilot console logs review * Refactor task scheduling to support tenant-level background work and enhance logging functionality. * Introduce ConsoleStreamHook for stdout/stderr tee and enhance logging validation. Adjust test cases and startup warnings for distributed lock provider usage. * Refactor console logging pipeline with capture optimization and new ConsoleLogsHost; update tests accordingly. * Add Ansi SGR parser for console logs and associated unit tests * Remove ANSI color renderings and parsers; integrate ConsoleLogScopeAccessor for improved logging context with workflow instance ID support. * Address console logs code quality feedback * Address PR review feedback * Preserve console logs extension points * Stabilize console logs host lifecycle * Address final automated review comments * Tighten console log capture shutdown * Address console log review feedback * Address follow-up review feedback * Cover final review feedback * Avoid recursive console provider initialization * Guard console host lease shutdown * Preserve console log scope and provider lifetime * Correlate console log scope fallback * Tighten console scope correlation * Expose host services during provider construction * Redact ANSI-normalized console lines * Add OpenTelemetry diagnostics backend foundation * Add OTLP HTTP ingestion parsing * Document OpenTelemetry diagnostics setup * Enforce OpenTelemetry hub permissions * Remove `ConsoleCaptureTee` and related services and tests * Add OpenTelemetry HTTP ingestion integration test * Use pipeline contributors for console log context * Update CShells package versions to 0.0.24-preview.132 * Add OpenTelemetry ingestion security tests * Add OpenTelemetry API authorization tests * Filter live console logs by workflow instance * Add OpenTelemetry hub tests * Add OpenTelemetry gRPC metadata hook * Assert OpenTelemetry workflow tags survive ingestion * Mark OpenTelemetry core build verified * Enhance console logging with activity execution metadata and extend test coverage. * Address console logs stream consumption comment * Wire OpenTelemetry diagnostics into core sample * Address Core diagnostics review feedback * Address Core Copilot follow-up feedback * Add OpenTelemetry metric instrument names * Address Core Copilot provider feedback * Address Core Copilot diagnostics follow-up * Address Core Copilot live feed feedback * Address Core Copilot store feedback * Integrate OpenTelemetry for logging, tracing, and metrics in ModularServer and update launch settings and docker-compose configuration. * Refactor to replace `ConsoleLogStream.Core` with `ConsoleLogStreaming.Core` across codebase and update `ConsoleStreamHook` installation. * Add diagnostics OpenTelemetry backend * Fix OpenTelemetry live hub subscription * Fix modular OpenTelemetry exporter endpoints * Add CShells logging configuration in appsettings.json * Remove obsolete unit tests and helper classes * Restore default activity exception handling * Simplify type serialization and alias management This commit refactors the internal type serialization and alias management system to reduce boilerplate, improve robustness, and simplify the developer experience: - Removed numerous explicit `ExpressionOptions` type alias registrations across various modules. - Updated `TypeJsonConverter` and polymorphic serialization to reliably handle types using assembly-qualified names when a short alias is not explicitly registered. - Streamlined `ExcludeFromHashConverter` to strictly adhere to `ExcludeFromHashAttribute` for hash calculations, removing complex `JsonIgnoreCondition` logic. - Eliminated several helper classes (`WorkflowJsonTypeResolver`, `WorkflowTypeValidator`, `IWorkflowTypeRegistry`, `WorkflowFactoryDictionary`, `JavaScriptExceptionTypeAliasRegistrar`, `WorkflowRuntimeTypeAliasRegistrar`) and their associated unit tests, simplifying the codebase. Additionally, this commit introduces a comprehensive markdown document (`product-website-feature-source.md`) outlining Elsa's core features, Studio capabilities, extension ecosystem, and architectural selling points, intended as source material for the product website. * Refine type serialization for improved robustness and alias handling This commit further enhances the type serialization and deserialization mechanisms: * Centralizes type resolution and alias management through `IWellKnownTypeRegistry` and `WorkflowJsonTypeResolver`. * Prioritizes registered type aliases when serializing type metadata in `PolymorphicObjectConverter`, resulting in more concise JSON output. * Enhances deserialization in `PolymorphicObjectConverter` and `VariableMapper` to gracefully handle unknown or non-instantiable types, providing fallbacks and logging warnings. * Simplifies `TypeJsonConverter` by delegating complex type resolution logic to the `WorkflowJsonTypeResolver`. * Adds `JsonArray` to the well-known type aliases for direct recognition. * Fix console logs packaging and workflow type resolution * Fix console log metadata and type resolution * Address Copilot review feedback * Enhance type resolution, improve console log handling, and update tests - Streamlined `WorkflowDictionaryExtensions` for better workflow registration validation. - Refined `ConsoleLogsAuthorizationTests` with the new `SetJsonRequest` helper to improve test requests handling. - Updated `OrderDefinition` to ignore JSON serialization for `KeySelector`. - Enhanced `WorkflowRuntimeFeature` for improved workflow registration and type alias configuration. - Added tests to ensure `ConsoleLogProvider` metadata filtration in various scenarios. - Improved type serialization logic in `WorkflowJsonTypeResolver`. - Updated README to fix references related to diagnostics. - Optimized `ExcludeFromHashConverter` for property serialization conditions. - Modified `TriggerIndexer` for streamlined trigger management. - Tested payload checks in `PublishEventTests`. - Adjusted `Endpoint` in `ConsoleLogs` for automatic JSON request handling. - Ensured registration of workflow type aliases in `WorkflowsFeature`. * Restore CLR workflow registration compatibility * Align JSON island serialization fixtures * Add Console Logs Services and Enhance Endpoint Handling - Introduced `ActivityExecutionsEndpointTests` to validate route exposure. - Added `ConsoleLogCaptureHostedService` for console log streaming. - Implemented `ConsoleStreamJsonConverter` for JSON conversion of console streams. - Developed `ElsaConsoleLogRecentBuffer` to handle recent log buffering. - Updated `ConsoleLogsAuthorizationTests` with new test cases for stream filter mapping. - Consolidated console log provider dependencies and registration, including recent buffering. - Enhanced `ElsaConsoleLogProvider` to use recent buffer for filtering. - Adjusted `Program.cs` for streamlined logging service setup. * Enhance type resolution and test coverage; streamline console log integration - Added `ConsoleStreamHook` for streamlined log streaming. - Updated `WorkflowJsonTypeResolverTests` to improve type resolution and test new scenarios. - Simplified type resolution by removing trusted assembly checks. * Fix CI smoke and package restore failures * Fix Docker smoke image project paths * Fix Docker Python runtime packages * Fix Docker CA smoke teardown * Refresh Elsa roadmap * Implement background processors and mediation coordination - Added `BackgroundCommandProcessor`, `BackgroundJobProcessor`, and `BackgroundNotificationProcessor` classes for handling commands, jobs, and notifications, respectively. - Introduced `MediatorBackgroundProcessingCoordinator` to coordinate the execution of all background processors. - Implemented `MediatorBackgroundTask` for wrapping `MediatorBackgroundProcessingCoordinator` in `BackgroundTask`. - Added unit tests for `MediatorBackgroundTask` to ensure proper start and stop behavior. - Refactored `BackgroundCommandSenderHostedService` to utilize `BackgroundCommandProcessor`. - Introduced 'elsa-roadmap-refresh' skill configuration for roadmap updates. * Address workflow type resolution review feedback * Address follow-up review feedback * Restore recent console logs execute path * Address Copilot follow-up review * Decouple workflow JSON aliases from expressions * Fix workflow management unit test setup * Fix console logs recent endpoint handler shape * Respect workflow JSON strict type aliases * Remove unused console log contracts reference * Address Copilot review feedback * Address Copilot follow-up comments * Synchronize ring buffer dropped count * Address background processor strategy replay
143 lines
5.6 KiB
C#
143 lines
5.6 KiB
C#
using System.Buffers.Binary;
|
|
using System.Text;
|
|
using Elsa.Diagnostics.OpenTelemetry.Ingestion.HttpProtobuf;
|
|
using Elsa.Diagnostics.OpenTelemetry.Models;
|
|
|
|
namespace Elsa.Diagnostics.OpenTelemetry.UnitTests.Ingestion;
|
|
|
|
public class OtlpHttpProtobufParserTests
|
|
{
|
|
private static readonly byte[] TraceId = Convert.FromHexString("00112233445566778899aabbccddeeff");
|
|
private static readonly byte[] SpanId = Convert.FromHexString("0011223344556677");
|
|
private static readonly DateTimeOffset Timestamp = new(2026, 5, 26, 10, 0, 0, TimeSpan.Zero);
|
|
|
|
[Fact(DisplayName = "OTLP trace payload is normalized to trace, span, resource, and workflow metadata")]
|
|
public void ParseTraces()
|
|
{
|
|
var payload = Message(1,
|
|
Join(Message(1, Resource()),
|
|
Message(2, Message(2,
|
|
Join(Bytes(1, TraceId),
|
|
Bytes(2, SpanId),
|
|
String(5, "Workflow/Approve"),
|
|
Varint(6, 1),
|
|
Varint(7, UnixNanos(Timestamp)),
|
|
Varint(8, UnixNanos(Timestamp.AddMilliseconds(25))),
|
|
Message(9, KeyValue("workflow.instance.id", "wf-1")),
|
|
Message(15, Varint(3, 2)))))));
|
|
|
|
var batch = OtlpHttpProtobufParser.ParseTraces(payload);
|
|
var span = Assert.Single(batch.Spans);
|
|
var trace = Assert.Single(batch.Traces);
|
|
|
|
Assert.Equal("elsa-server:node-1", span.ResourceId);
|
|
Assert.Equal("00112233445566778899aabbccddeeff", trace.TraceId);
|
|
Assert.Equal("0011223344556677", trace.RootSpanId);
|
|
Assert.Equal("wf-1", Assert.Single(trace.WorkflowInstanceIds));
|
|
Assert.Equal(SpanStatus.Error, trace.Status);
|
|
}
|
|
|
|
[Fact(DisplayName = "OTLP metric payload is normalized to metric instruments and points")]
|
|
public void ParseMetrics()
|
|
{
|
|
var point =
|
|
Join(Varint(3, UnixNanos(Timestamp)),
|
|
Fixed64(4, 42.5),
|
|
Message(7, KeyValue("workflow.definition.id", "orders")));
|
|
var metric =
|
|
Join(String(1, "workflow.duration"),
|
|
String(2, "Workflow duration"),
|
|
String(3, "ms"),
|
|
Message(5, Message(1, point)));
|
|
var payload = Message(1, Join(Message(1, Resource()), Message(2, Message(2, metric))));
|
|
|
|
var batch = OtlpHttpProtobufParser.ParseMetrics(payload);
|
|
var instrument = Assert.Single(batch.Instruments);
|
|
var metricPoint = Assert.Single(batch.MetricPoints);
|
|
|
|
Assert.Equal("workflow.duration", instrument.Name);
|
|
Assert.Equal("workflow.duration", metricPoint.InstrumentName);
|
|
Assert.Equal(MetricKind.Gauge, instrument.Kind);
|
|
Assert.Equal(42.5, metricPoint.Value);
|
|
Assert.Equal("orders", metricPoint.Attributes["workflow.definition.id"]);
|
|
}
|
|
|
|
[Fact(DisplayName = "OTLP log payload is normalized to correlated OTLP log records")]
|
|
public void ParseLogs()
|
|
{
|
|
var log =
|
|
Join(Varint(1, UnixNanos(Timestamp)),
|
|
Varint(2, 17),
|
|
String(3, "Error"),
|
|
Message(5, AnyString("boom")),
|
|
Bytes(9, TraceId),
|
|
Bytes(10, SpanId),
|
|
Message(6, KeyValue("workflow.instance.id", "wf-1")));
|
|
var payload = Message(1, Join(Message(1, Resource()), Message(2, Message(2, log))));
|
|
|
|
var batch = OtlpHttpProtobufParser.ParseLogs(payload);
|
|
var record = Assert.Single(batch.Logs);
|
|
|
|
Assert.Equal("elsa-server:node-1", record.ResourceId);
|
|
Assert.Equal("Error", record.SeverityText);
|
|
Assert.Equal("boom", record.Body);
|
|
Assert.Equal("00112233445566778899aabbccddeeff", record.TraceId);
|
|
Assert.Equal("0011223344556677", record.SpanId);
|
|
}
|
|
|
|
private static byte[] Resource()
|
|
{
|
|
return Join(
|
|
Message(1, KeyValue("service.name", "elsa-server")),
|
|
Message(1, KeyValue("service.instance.id", "node-1")),
|
|
Message(1, KeyValue("telemetry.sdk.language", "dotnet")));
|
|
}
|
|
|
|
private static byte[] KeyValue(string key, string value) => Join(String(1, key), Message(2, AnyString(value)));
|
|
|
|
private static byte[] AnyString(string value) => String(1, value);
|
|
|
|
private static byte[] Message(int fieldNumber, byte[] value) => Join(Varint((ulong)((fieldNumber << 3) | 2)), Varint((ulong)value.Length), value);
|
|
|
|
private static byte[] String(int fieldNumber, string value) => Message(fieldNumber, Encoding.UTF8.GetBytes(value));
|
|
|
|
private static byte[] Bytes(int fieldNumber, byte[] value) => Message(fieldNumber, value);
|
|
|
|
private static byte[] Varint(int fieldNumber, ulong value) => Join(Varint((ulong)(fieldNumber << 3)), Varint(value));
|
|
|
|
private static byte[] Fixed64(int fieldNumber, double value)
|
|
{
|
|
var bytes = new byte[9];
|
|
bytes[0] = (byte)((fieldNumber << 3) | 1);
|
|
BinaryPrimitives.WriteUInt64LittleEndian(bytes.AsSpan(1), (ulong)BitConverter.DoubleToInt64Bits(value));
|
|
return bytes;
|
|
}
|
|
|
|
private static byte[] Varint(ulong value)
|
|
{
|
|
var bytes = new List<byte>();
|
|
while (value >= 0x80)
|
|
{
|
|
bytes.Add((byte)(value | 0x80));
|
|
value >>= 7;
|
|
}
|
|
bytes.Add((byte)value);
|
|
return bytes.ToArray();
|
|
}
|
|
|
|
private static ulong UnixNanos(DateTimeOffset timestamp) => (ulong)(timestamp - DateTimeOffset.UnixEpoch).Ticks * 100;
|
|
|
|
private static byte[] Join(params byte[][] segments)
|
|
{
|
|
var result = new byte[segments.Sum(x => x.Length)];
|
|
var offset = 0;
|
|
foreach (var segment in segments)
|
|
{
|
|
segment.CopyTo(result, offset);
|
|
offset += segment.Length;
|
|
}
|
|
|
|
return result;
|
|
}
|
|
}
|