Commit graph

9 commits

Author SHA1 Message Date
Sipke Schoorstra c2fb027c41
Refactor: Overhauls workflow JSON type serialization (#7549)
* Avoid null endpoint DTO metadata in tests

* Enforce console logs hub read permission

* Remove unused console logs hub import

* Support mapped endpoint metadata in auth tests

* Reduce console log capture throughput impact

* Address Copilot console logs review

* Refactor task scheduling to support tenant-level background work and enhance logging functionality.

* Introduce ConsoleStreamHook for stdout/stderr tee and enhance logging validation. Adjust test cases and startup warnings for distributed lock provider usage.

* Refactor console logging pipeline with capture optimization and new ConsoleLogsHost; update tests accordingly.

* Add Ansi SGR parser for console logs and associated unit tests

* Remove ANSI color renderings and parsers; integrate ConsoleLogScopeAccessor for improved logging context with workflow instance ID support.

* Address console logs code quality feedback

* Address PR review feedback

* Preserve console logs extension points

* Stabilize console logs host lifecycle

* Address final automated review comments

* Tighten console log capture shutdown

* Address console log review feedback

* Address follow-up review feedback

* Cover final review feedback

* Avoid recursive console provider initialization

* Guard console host lease shutdown

* Preserve console log scope and provider lifetime

* Correlate console log scope fallback

* Tighten console scope correlation

* Expose host services during provider construction

* Redact ANSI-normalized console lines

* Remove `ConsoleCaptureTee` and related services and tests

* Use pipeline contributors for console log context

* Update CShells package versions to 0.0.24-preview.132

* Filter live console logs by workflow instance

* Enhance console logging with activity execution metadata and extend test coverage.

* Address console logs stream consumption comment

* Add diagnostics OpenTelemetry backend

* Introduce dedicated workflow JSON type registry and hardening

This change addresses GitHub issue #7541 by establishing a separate type registry (`IWorkflowJsonTypeRegistry`) for workflow JSON serialization. This decouples workflow type resolution from expression type aliases, enforcing a strict trust boundary.

Key aspects:
- New workflow JSON emits preferred aliases for registered types.
- Existing persisted workflows can be loaded via registered legacy names.
- Unknown, abstract, interface, open generic, or inappropriate collection types are rejected during deserialization, enhancing security.
- Public APIs (e.g., incident strategies) now expose consistent workflow JSON type identifiers.

This ensures secure, predictable, and backward-compatible handling of types within workflow definitions and payloads.

* Remove unused project references and streamline console log endpoint

* Move serialization type aliases to Elsa.Common

* Update serialization integration fixtures for aliases

* Stabilize missing rate limiter policy test
2026-05-31 11:09:39 +02:00
Sipke Schoorstra 842cf7c162
[codex] Fix console log metadata and type resolution (#7542)
* Avoid null endpoint DTO metadata in tests

* Enforce console logs hub read permission

* Remove unused console logs hub import

* Support mapped endpoint metadata in auth tests

* Reduce console log capture throughput impact

* Address Copilot console logs review

* Refactor task scheduling to support tenant-level background work and enhance logging functionality.

* Introduce ConsoleStreamHook for stdout/stderr tee and enhance logging validation. Adjust test cases and startup warnings for distributed lock provider usage.

* Refactor console logging pipeline with capture optimization and new ConsoleLogsHost; update tests accordingly.

* Add Ansi SGR parser for console logs and associated unit tests

* Remove ANSI color renderings and parsers; integrate ConsoleLogScopeAccessor for improved logging context with workflow instance ID support.

* Address console logs code quality feedback

* Address PR review feedback

* Preserve console logs extension points

* Stabilize console logs host lifecycle

* Address final automated review comments

* Tighten console log capture shutdown

* Address console log review feedback

* Address follow-up review feedback

* Cover final review feedback

* Avoid recursive console provider initialization

* Guard console host lease shutdown

* Preserve console log scope and provider lifetime

* Correlate console log scope fallback

* Tighten console scope correlation

* Expose host services during provider construction

* Redact ANSI-normalized console lines

* Add OpenTelemetry diagnostics backend foundation

* Add OTLP HTTP ingestion parsing

* Document OpenTelemetry diagnostics setup

* Enforce OpenTelemetry hub permissions

* Remove `ConsoleCaptureTee` and related services and tests

* Add OpenTelemetry HTTP ingestion integration test

* Use pipeline contributors for console log context

* Update CShells package versions to 0.0.24-preview.132

* Add OpenTelemetry ingestion security tests

* Add OpenTelemetry API authorization tests

* Filter live console logs by workflow instance

* Add OpenTelemetry hub tests

* Add OpenTelemetry gRPC metadata hook

* Assert OpenTelemetry workflow tags survive ingestion

* Mark OpenTelemetry core build verified

* Enhance console logging with activity execution metadata and extend test coverage.

* Address console logs stream consumption comment

* Wire OpenTelemetry diagnostics into core sample

* Address Core diagnostics review feedback

* Address Core Copilot follow-up feedback

* Add OpenTelemetry metric instrument names

* Address Core Copilot provider feedback

* Address Core Copilot diagnostics follow-up

* Address Core Copilot live feed feedback

* Address Core Copilot store feedback

* Integrate OpenTelemetry for logging, tracing, and metrics in ModularServer and update launch settings and docker-compose configuration.

* Refactor to replace `ConsoleLogStream.Core` with `ConsoleLogStreaming.Core` across codebase and update `ConsoleStreamHook` installation.

* Add diagnostics OpenTelemetry backend

* Fix OpenTelemetry live hub subscription

* Fix modular OpenTelemetry exporter endpoints

* Add CShells logging configuration in appsettings.json

* Remove obsolete unit tests and helper classes

* Restore default activity exception handling

* Simplify type serialization and alias management

This commit refactors the internal type serialization and alias management system to reduce boilerplate, improve robustness, and simplify the developer experience:

-   Removed numerous explicit `ExpressionOptions` type alias registrations across various modules.
-   Updated `TypeJsonConverter` and polymorphic serialization to reliably handle types using assembly-qualified names when a short alias is not explicitly registered.
-   Streamlined `ExcludeFromHashConverter` to strictly adhere to `ExcludeFromHashAttribute` for hash calculations, removing complex `JsonIgnoreCondition` logic.
-   Eliminated several helper classes (`WorkflowJsonTypeResolver`, `WorkflowTypeValidator`, `IWorkflowTypeRegistry`, `WorkflowFactoryDictionary`, `JavaScriptExceptionTypeAliasRegistrar`, `WorkflowRuntimeTypeAliasRegistrar`) and their associated unit tests, simplifying the codebase.

Additionally, this commit introduces a comprehensive markdown document (`product-website-feature-source.md`) outlining Elsa's core features, Studio capabilities, extension ecosystem, and architectural selling points, intended as source material for the product website.

* Refine type serialization for improved robustness and alias handling

This commit further enhances the type serialization and deserialization mechanisms:

*   Centralizes type resolution and alias management through `IWellKnownTypeRegistry` and `WorkflowJsonTypeResolver`.
*   Prioritizes registered type aliases when serializing type metadata in `PolymorphicObjectConverter`, resulting in more concise JSON output.
*   Enhances deserialization in `PolymorphicObjectConverter` and `VariableMapper` to gracefully handle unknown or non-instantiable types, providing fallbacks and logging warnings.
*   Simplifies `TypeJsonConverter` by delegating complex type resolution logic to the `WorkflowJsonTypeResolver`.
*   Adds `JsonArray` to the well-known type aliases for direct recognition.

* Fix console logs packaging and workflow type resolution

* Fix console log metadata and type resolution

* Address Copilot review feedback

* Enhance type resolution, improve console log handling, and update tests

- Streamlined `WorkflowDictionaryExtensions` for better workflow registration validation.
- Refined `ConsoleLogsAuthorizationTests` with the new `SetJsonRequest` helper to improve test requests handling.
- Updated `OrderDefinition` to ignore JSON serialization for `KeySelector`.
- Enhanced `WorkflowRuntimeFeature` for improved workflow registration and type alias configuration.
- Added tests to ensure `ConsoleLogProvider` metadata filtration in various scenarios.
- Improved type serialization logic in `WorkflowJsonTypeResolver`.
- Updated README to fix references related to diagnostics.
- Optimized `ExcludeFromHashConverter` for property serialization conditions.
- Modified `TriggerIndexer` for streamlined trigger management.
- Tested payload checks in `PublishEventTests`.
- Adjusted `Endpoint` in `ConsoleLogs` for automatic JSON request handling.
- Ensured registration of workflow type aliases in `WorkflowsFeature`.

* Restore CLR workflow registration compatibility

* Align JSON island serialization fixtures

* Add Console Logs Services and Enhance Endpoint Handling

- Introduced `ActivityExecutionsEndpointTests` to validate route exposure.
- Added `ConsoleLogCaptureHostedService` for console log streaming.
- Implemented `ConsoleStreamJsonConverter` for JSON conversion of console streams.
- Developed `ElsaConsoleLogRecentBuffer` to handle recent log buffering.
- Updated `ConsoleLogsAuthorizationTests` with new test cases for stream filter mapping.
- Consolidated console log provider dependencies and registration, including recent buffering.
- Enhanced `ElsaConsoleLogProvider` to use recent buffer for filtering.
- Adjusted `Program.cs` for streamlined logging service setup.

* Enhance type resolution and test coverage; streamline console log integration

- Added `ConsoleStreamHook` for streamlined log streaming.
- Updated `WorkflowJsonTypeResolverTests` to improve type resolution and test new scenarios.
- Simplified type resolution by removing trusted assembly checks.

* Fix CI smoke and package restore failures

* Fix Docker smoke image project paths

* Fix Docker Python runtime packages

* Fix Docker CA smoke teardown

* Refresh Elsa roadmap

* Implement background processors and mediation coordination

- Added `BackgroundCommandProcessor`, `BackgroundJobProcessor`, and `BackgroundNotificationProcessor` classes for handling commands, jobs, and notifications, respectively.
- Introduced `MediatorBackgroundProcessingCoordinator` to coordinate the execution of all background processors.
- Implemented `MediatorBackgroundTask` for wrapping `MediatorBackgroundProcessingCoordinator` in `BackgroundTask`.
- Added unit tests for `MediatorBackgroundTask` to ensure proper start and stop behavior.
- Refactored `BackgroundCommandSenderHostedService` to utilize `BackgroundCommandProcessor`.
- Introduced 'elsa-roadmap-refresh' skill configuration for roadmap updates.

* Address workflow type resolution review feedback

* Address follow-up review feedback

* Restore recent console logs execute path

* Address Copilot follow-up review

* Decouple workflow JSON aliases from expressions

* Fix workflow management unit test setup

* Fix console logs recent endpoint handler shape

* Respect workflow JSON strict type aliases

* Remove unused console log contracts reference

* Address Copilot review feedback

* Address Copilot follow-up comments

* Synchronize ring buffer dropped count

* Address background processor strategy replay
2026-05-30 22:52:01 +02:00
Sipke Schoorstra e2e00ff235
Add secrets module (#7468)
* Add secrets module

* Address Greptile feedback for secrets module

* Handle unavailable secrets in provider adapter

* Address path combine review comments

* Address additional Greptile secrets review

* Address final Greptile secrets feedback

* Handle secrets test payload failures

* address greptile feedback on secrets rotation

* fix secret recreation concurrency

* address greptile secrets followups

* address greptile secrets reliability feedback

* align secret store capabilities
2026-05-20 11:48:01 +02:00
Sipke Schoorstra 43108c2e48
Add diagnostics console logs (#7462)
* feat: add diagnostics console logs

* test: avoid secret-like redaction fixtures

* fix: address console logs review feedback

* fix: harden console log capture lifecycle

* fix: report console log drop summaries

* fix: address diagnostics review cleanups
2026-05-18 02:16:46 +02:00
Sipke Schoorstra 6485f05a87
Add state machine activity (#7457)
* Add state machine activity

* address greptile state machine feedback

* address state machine trigger cancellation feedback
2026-05-18 02:09:03 +02:00
Sipke Schoorstra 827ad6bc6b
[codex] Add structured log SQLite persistence (#7445)
* Add structured log persistence spec

* Clarify structured log persistence spec

* Plan structured log persistence implementation

* Regenerate structured log persistence tasks

* Address structured log persistence analysis findings

* Add structured log SQLite persistence

* Address structured log persistence review

* Harden structured log write buffer shutdown

* Start structured log SQLite migrations before buffer
2026-05-13 15:57:06 +02:00
Sipke Schoorstra ab3e46bbe2
[codex] Add live server log streaming diagnostics (#7438)
* Add live server logs Spec Kit plan

* Implement live server logs diagnostics module

* Add server log sources and redaction hardening

* Add diagnostics unit tests

* Harden server log hub subscriptions

* Secure server log hub permissions

* Validate server log filter updates

* Add diagnostics logger and source tests

* Add diagnostics integration test project

* Add multi-source diagnostics provider coverage

* Broadcast server log source changes

* Document diagnostics server log streaming

* Add diagnostics sample host wiring

* Record diagnostics validation results

* Address server log PR feedback

* Rename diagnostics module to server logs

* Add server logs shell feature

* Make server logs shell options bindable

* Accept read wildcard for server logs

* Align server logs authorization with API patterns

* Update CShells structure and logging levels, add diagnostics module

* Rename PostgreSql shell feature classes for consistency

* Switch from Sqlite to PostgreSQL for workflow and identity persistence, add QuartzPostgreSql configuration

* Refactor server logs into diagnostics structured logs (#7440)

* Specify diagnostics structured logs refactor

* docs: clarify structured logs spec

* docs: plan diagnostics structured logs

* docs: add diagnostics structured logs tasks

* refactor: rename server logs to diagnostics structured logs

* Refactor PostgreSql persistence features to use centralized entity model handler registration.

* Refactor EFCore persistence features to centralize entity model handler registration for MySql, Sqlite, and Oracle providers.

* Integrate structured logs by renaming server logs, adjusting appsettings, and updating project references.

* Switch from PostgreSQL to Sqlite for workflow and identity persistence, update appsettings configuration.
2026-05-11 00:08:52 +02:00
Sipke Schoorstra 3e43b85a2b
docs: add agent operating principles to AGENTS.md and CLAUDE.md 2026-05-06 12:44:27 +02:00
Sipke Schoorstra 80d3ad2b47
docs: add agent contributor guidance 2026-05-05 15:43:01 +02:00