* Avoid null endpoint DTO metadata in tests
* Enforce console logs hub read permission
* Remove unused console logs hub import
* Support mapped endpoint metadata in auth tests
* Reduce console log capture throughput impact
* Address Copilot console logs review
* Refactor task scheduling to support tenant-level background work and enhance logging functionality.
* Introduce ConsoleStreamHook for stdout/stderr tee and enhance logging validation. Adjust test cases and startup warnings for distributed lock provider usage.
* Refactor console logging pipeline with capture optimization and new ConsoleLogsHost; update tests accordingly.
* Add Ansi SGR parser for console logs and associated unit tests
* Remove ANSI color renderings and parsers; integrate ConsoleLogScopeAccessor for improved logging context with workflow instance ID support.
* Address console logs code quality feedback
* Address PR review feedback
* Preserve console logs extension points
* Stabilize console logs host lifecycle
* Address final automated review comments
* Tighten console log capture shutdown
* Address console log review feedback
* Address follow-up review feedback
* Cover final review feedback
* Avoid recursive console provider initialization
* Guard console host lease shutdown
* Preserve console log scope and provider lifetime
* Correlate console log scope fallback
* Tighten console scope correlation
* Expose host services during provider construction
* Redact ANSI-normalized console lines
* Add OpenTelemetry diagnostics backend foundation
* Add OTLP HTTP ingestion parsing
* Document OpenTelemetry diagnostics setup
* Enforce OpenTelemetry hub permissions
* Remove `ConsoleCaptureTee` and related services and tests
* Add OpenTelemetry HTTP ingestion integration test
* Use pipeline contributors for console log context
* Update CShells package versions to 0.0.24-preview.132
* Add OpenTelemetry ingestion security tests
* Add OpenTelemetry API authorization tests
* Filter live console logs by workflow instance
* Add OpenTelemetry hub tests
* Add OpenTelemetry gRPC metadata hook
* Assert OpenTelemetry workflow tags survive ingestion
* Mark OpenTelemetry core build verified
* Enhance console logging with activity execution metadata and extend test coverage.
* Address console logs stream consumption comment
* Wire OpenTelemetry diagnostics into core sample
* Address Core diagnostics review feedback
* Address Core Copilot follow-up feedback
* Add OpenTelemetry metric instrument names
* Address Core Copilot provider feedback
* Address Core Copilot diagnostics follow-up
* Address Core Copilot live feed feedback
* Address Core Copilot store feedback
* Integrate OpenTelemetry for logging, tracing, and metrics in ModularServer and update launch settings and docker-compose configuration.
* Refactor to replace `ConsoleLogStream.Core` with `ConsoleLogStreaming.Core` across codebase and update `ConsoleStreamHook` installation.
* Add diagnostics OpenTelemetry backend
* Fix OpenTelemetry live hub subscription
* Fix modular OpenTelemetry exporter endpoints
* Add CShells logging configuration in appsettings.json
* Remove obsolete unit tests and helper classes
* Restore default activity exception handling
* Simplify type serialization and alias management
This commit refactors the internal type serialization and alias management system to reduce boilerplate, improve robustness, and simplify the developer experience:
- Removed numerous explicit `ExpressionOptions` type alias registrations across various modules.
- Updated `TypeJsonConverter` and polymorphic serialization to reliably handle types using assembly-qualified names when a short alias is not explicitly registered.
- Streamlined `ExcludeFromHashConverter` to strictly adhere to `ExcludeFromHashAttribute` for hash calculations, removing complex `JsonIgnoreCondition` logic.
- Eliminated several helper classes (`WorkflowJsonTypeResolver`, `WorkflowTypeValidator`, `IWorkflowTypeRegistry`, `WorkflowFactoryDictionary`, `JavaScriptExceptionTypeAliasRegistrar`, `WorkflowRuntimeTypeAliasRegistrar`) and their associated unit tests, simplifying the codebase.
Additionally, this commit introduces a comprehensive markdown document (`product-website-feature-source.md`) outlining Elsa's core features, Studio capabilities, extension ecosystem, and architectural selling points, intended as source material for the product website.
* Refine type serialization for improved robustness and alias handling
This commit further enhances the type serialization and deserialization mechanisms:
* Centralizes type resolution and alias management through `IWellKnownTypeRegistry` and `WorkflowJsonTypeResolver`.
* Prioritizes registered type aliases when serializing type metadata in `PolymorphicObjectConverter`, resulting in more concise JSON output.
* Enhances deserialization in `PolymorphicObjectConverter` and `VariableMapper` to gracefully handle unknown or non-instantiable types, providing fallbacks and logging warnings.
* Simplifies `TypeJsonConverter` by delegating complex type resolution logic to the `WorkflowJsonTypeResolver`.
* Adds `JsonArray` to the well-known type aliases for direct recognition.
* Fix console logs packaging and workflow type resolution
* Fix console log metadata and type resolution
* Address Copilot review feedback
* Enhance type resolution, improve console log handling, and update tests
- Streamlined `WorkflowDictionaryExtensions` for better workflow registration validation.
- Refined `ConsoleLogsAuthorizationTests` with the new `SetJsonRequest` helper to improve test requests handling.
- Updated `OrderDefinition` to ignore JSON serialization for `KeySelector`.
- Enhanced `WorkflowRuntimeFeature` for improved workflow registration and type alias configuration.
- Added tests to ensure `ConsoleLogProvider` metadata filtration in various scenarios.
- Improved type serialization logic in `WorkflowJsonTypeResolver`.
- Updated README to fix references related to diagnostics.
- Optimized `ExcludeFromHashConverter` for property serialization conditions.
- Modified `TriggerIndexer` for streamlined trigger management.
- Tested payload checks in `PublishEventTests`.
- Adjusted `Endpoint` in `ConsoleLogs` for automatic JSON request handling.
- Ensured registration of workflow type aliases in `WorkflowsFeature`.
* Restore CLR workflow registration compatibility
* Align JSON island serialization fixtures
* Add Console Logs Services and Enhance Endpoint Handling
- Introduced `ActivityExecutionsEndpointTests` to validate route exposure.
- Added `ConsoleLogCaptureHostedService` for console log streaming.
- Implemented `ConsoleStreamJsonConverter` for JSON conversion of console streams.
- Developed `ElsaConsoleLogRecentBuffer` to handle recent log buffering.
- Updated `ConsoleLogsAuthorizationTests` with new test cases for stream filter mapping.
- Consolidated console log provider dependencies and registration, including recent buffering.
- Enhanced `ElsaConsoleLogProvider` to use recent buffer for filtering.
- Adjusted `Program.cs` for streamlined logging service setup.
* Enhance type resolution and test coverage; streamline console log integration
- Added `ConsoleStreamHook` for streamlined log streaming.
- Updated `WorkflowJsonTypeResolverTests` to improve type resolution and test new scenarios.
- Simplified type resolution by removing trusted assembly checks.
* Fix CI smoke and package restore failures
* Fix Docker smoke image project paths
* Fix Docker Python runtime packages
* Fix Docker CA smoke teardown
* Refresh Elsa roadmap
* Implement background processors and mediation coordination
- Added `BackgroundCommandProcessor`, `BackgroundJobProcessor`, and `BackgroundNotificationProcessor` classes for handling commands, jobs, and notifications, respectively.
- Introduced `MediatorBackgroundProcessingCoordinator` to coordinate the execution of all background processors.
- Implemented `MediatorBackgroundTask` for wrapping `MediatorBackgroundProcessingCoordinator` in `BackgroundTask`.
- Added unit tests for `MediatorBackgroundTask` to ensure proper start and stop behavior.
- Refactored `BackgroundCommandSenderHostedService` to utilize `BackgroundCommandProcessor`.
- Introduced 'elsa-roadmap-refresh' skill configuration for roadmap updates.
* Address workflow type resolution review feedback
* Address follow-up review feedback
* Restore recent console logs execute path
* Address Copilot follow-up review
* Decouple workflow JSON aliases from expressions
* Fix workflow management unit test setup
* Fix console logs recent endpoint handler shape
* Respect workflow JSON strict type aliases
* Remove unused console log contracts reference
* Address Copilot review feedback
* Address Copilot follow-up comments
* Synchronize ring buffer dropped count
* Address background processor strategy replay
Removed unnecessary Docker services, configuration files, and volumes from `docker-compose.yml`. Cleaned up unused project references in `Elsa.sln`. Streamlined local development setup by eliminating redundant resources.
* Introduce `IWorkflowResumer` and `ActivityInputEvaluatorContext`, refactor endpoint handling, extend logging, and improve bookmark queue processing.
* Remove deprecated WorkflowContexts module and optimize project.
Deleted the Elsa.Studio.WorkflowContexts module and references from solution files. Corrected minor errors in remaining code and updated project configurations to align with the new structure.```
* Update GitHub workflows to track `develop/3.6.0` branch instead of `patch/3.5.1`.
* Fix inconsistent formatting in `InputDescriptor` constructor and properties.
* Add XML documentation for `DictionaryValueEvaluator` in `UIHints/Dictionary` module
* Refactor `DictionaryValueEvaluator` to improve readability and simplify dictionary evaluation logic.
Integrated Azurite for local Azure Storage emulation, including Blob, Queue, and Table services. Configured ports, volumes, and debug options for development purposes. This addition aims to enhance local testing and reduce dependencies on external Azure services.
Introduced an SQL Server service using the official 2022 image. Configured environment variables, ports, and a volume for persistent data storage. This addition supports development and testing scenarios requiring SQL Server.
Moved Oracle setup scripts to a more intuitive directory (`docker/setup`) and updated corresponding volume mappings in `docker-compose.yml`. Adjusted `.gitignore` to exclude the new `docker/data/` directory. This change improves organization and clarity for Oracle-related files.
Revised database initialization scripts to better handle Postgres and Oracle environments, introduced schema-specific configurations for Oracle EFCore, and cleaned up obsolete or redundant entity model setup. Streamlined project structure by relocating and renaming files for SQLite and Oracle EFCore setups, improving maintainability and readability.
This commit introduces Oracle database integration by adding necessary configurations, entity mappings, and Docker Compose setup. Oracle-specific mappings ensure compatibility with NCLOB for large data handling. The changes also include updates to appsettings, enum for SqlDatabaseProvider, and project references to support Oracle.
Add support for EF Core primitive collections in MySQL by specifying `EnablePrimitiveCollectionsSupport()` in the DbContext configuration. Updated the Docker Compose file to rename the MySQL volume and adjusted appsettings.json to include a MySQL connection string and set MySQL as the default database provider.
Upgraded RabbitMQ from version 3-management to 4-management across tests and the Docker setup. This ensures compatibility with the latest features and improvements in RabbitMQ while maintaining consistency across environments.
* Add Kafka module with integration and example setup
Introduced the Kafka module providing consumer integration and activities into the project. This includes new classes for consumer handling, configuration, and activities. An example setup using Docker Compose is also added to facilitate development and testing.
* Refactor KafkaTransportMessage to inline Timestamp namespace
Simplify the namespace usage for the Timestamp type within the KafkaTransportMessage record. This change eliminates the need for a separate using directive for Timestamp, enhancing code readability and maintainability.
* Add support for handling Kafka transport messages
This commit introduces the capability to handle and trigger workflows based on Kafka transport messages. It adds a new handler, notifications, and updates the message stimulus to include correlating fields. Additionally, the Kafka consumers are now managed more modularly with updated startup tasks and mediator integration.
* Enable Kafka integration and fix Kafka options naming
Added support for Kafka integration in Elsa.Server.Web by setting up Kafka configurations in appsettings.json and updating Program.cs. Also, renamed `ConsumerConfigs` to `ConsumerDefinitions` in Kafka options for clarity.
* Add consumer definition enumeration and dropdown support
Introduced `IConsumerDefinitionEnumerator` for managing consumer definitions across providers and implemented in `ConsumerDefinitionEnumerator` class. Enhanced `KafkaFeature` to register these services and updated the `MessageReceived` activity to use a dropdown UI hint for consuming definitions. Improved `StartConsumersTask` by refactoring consumer definition retrieval logic.
* Add SendMessage activity and refine Kafka messaging
Introduce a new SendMessage activity for Kafka, enabling message publishing to specific topics. Refine KafkaTransportMessage model by removing headers and timestamp fields. Adjust the StimulusSender logic to streamline the bookmark queuing process and fix key-value pairing in dropdown options. Update appsettings for corrected Kafka bootstrap server and topic configurations.
* Add producer and topic management support
Introduced interfaces and implementations for managing producer and topic definitions along with their respective enumerators and list providers. Updated `SendMessage` activity to include producer selection and refactored consumer definition providers for better consistency.
* Refactor Kafka configuration property names
Renamed Kafka configuration properties for better consistency and readability across the codebase. Updated property names from `ProducerDefinitions` to `Producers`, `ConsumerDefinitions` to `Consumers`, and `TopicDefinitions` to `Topics`. Added missing input attribute in `SendMessage.cs` and registered additional handlers in `KafkaFeature.cs`.
* Add custom serializers for Kafka message handling
Introduced `DefaultSerializers` class for custom serialization and deserialization of Kafka messages. Updated `MessageReceived` and `SendMessage` activities to use these custom serializers, and modified `KafkaOptions` to include them.
* Fix ExpandoObject serialization method parameter
Changed the serialization type from `ExpandoObject` to the actual type of the object to ensure proper serialization. This ensures that derived types are correctly handled during the serialization process.
* Add Producer and Consumer workflows for Kafka
Introduced two new workflows: `ProducerWorkflow` and `ConsumerWorkflow` for handling Kafka messages. Updated `DefaultSerializers` to use camelCase property naming and modified `appsettings.json` to include `topic-2` and format entries.
* Add JSON serialization to log output in ConsumerWorkflow
This change enhances the log output by serializing messages to JSON format before writing them. The addition of System.Text.Json ensures that the message content is presented in a structured and standardized format in logs.
* Add correlation strategies and update Kafka features
Implemented HeaderCorrelationStrategy and NullCorrelationStrategy, and updated KafkaFeature to support customizable correlation strategies. Added correlation ID handling to Kafka transport messages and updated config and handlers accordingly.
* Add tenant accessor to ConsumerDefinitionWorkflowContextProvider
Integrated ITenantAccessor to the provider to support tenant-specific context loading. Updated the constructor and LoadAsync method to retrieve the tenant information and use it for context-specific operations.
* Switch to MySQL and disable Kafka
This commit changes the SQL database provider from SQLite to MySQL and disables Kafka use. It also includes necessary adjustments such as adding MySQL handling in configuration and connection setups, updating `docker-compose` to include MySQL services, and referencing MySQL projects in the `.csproj` file.
* Add UI property handlers to multiple features
This commit introduces various UI property handlers across several features such as Python, JavaScript, CSharp, and Workflow features to enhance user interface property handling. It also updates the property UI handler resolution logic to better manage cases where providers are not available. Furthermore, adjustments were made in the server configuration to switch database providers and enable Kafka.
* Refactor property UI handler retrieval logic
Modified the logic to fetch property UI handlers by preloading them into a list and then filtering. This change improves readability and potentially performance by reducing repetitive service provider calls.
* Incremental work on Kafka workers and predicate evaluation
* Merge BookmarkInvoker with BookmarkResumer
* Register IWorkerManager
* Change lifetime scope of WorkerManager to Singleton
* **Introduce topic subscription handling for Kafka workers**
Added `IWorkerTopicSubscriber` interface and its implementation for managing topic subscriptions. Enhanced workers to bind triggers and bookmarks dynamically based on existing data. Updated several classes and methods to support topic-based subscriptions and headers.
* Refactor trigger matching logic.
Extract trigger matching conditions into `IsMatchAsync` method for reuse. This enhances code maintainability and readability by reducing redundancy. The new `GetTopic` helper method isolates the topic retrieval logic.
* Add Name property to MassTransitActivityTypeProvider
This commit inserts the Name property in the returned object within the MassTransitActivityTypeProvider class. It ensures that the typeName is included, providing a clearer definition of the activity type.
* Add handling for deleted bookmarks and refactor bookmark removal
Added a new event handler for `BookmarksDeleted` to ensure removed bookmarks are processed correctly. Refactored the bookmark removal logic into a helper method to reduce code duplication and streamline the workflow.
* Switch to asynchronous bookmark queue processing
Refactored the `TriggerWorkflows` handler to use `IBookmarkQueue` instead of directly invoking the `IBookmarkResumer`. This change aims to improve scalability by queueing bookmark resumption requests, enabling better load distribution and async processing. Added necessary helpers and configuration options to support this functionality.
* Remove unused IBookmarkResumer dependency
Simplify the constructor by removing the unused IBookmarkResumer dependency. This cleanup reduces potential confusion and improves code maintainability without impacting functionality.
* Add support for local message processing
Introduced an `IsLocal` property to `MessageReceivedStimulus` for determining if the message event is local to a specific workflow instance. Updated `BookmarkBinding` and related handler methods to utilize `CorrelationId` for local event matching. Removed unused `CorrelatingFields` from `MessageReceived` activity.
* Add nullability checks to IWorker retrieval methods
Updated `GetWorker` methods to return nullable `IWorker` to handle cases where a worker might not exist. Modified code to include null checks and conditional operations to prevent potential null reference exceptions when accessing worker methods.
* Add filtering based on activity type name for triggers and bookmarks
This commit introduces filtering for triggers and bookmarks based on the `MessageReceived` activity type name. It also adds an option to mark messages as local in the `SendMessage` activity, where local messages are delivered to the current workflow instance only. These changes help enhance the management and targeted delivery of messages within the workflow framework.
* Add new Kafka topics and clean up producers config
New topics "topic-3" and "topic-4" were added to the Kafka settings. Unused topic references were removed from the producers configuration to simplify and improve clarity.
* Add predicate to KafkaConsumerActivity in ConsumerWorkflow
Introduced a predicate to the KafkaConsumerActivity using JavaScript expressions to filter messages based on OrderId. This ensures only relevant messages are processed in the workflow.
Introduce OpenTelemetry configuration and dependencies for observability. Replace DataDog tracer with OpenTelemetry auto-instrumentation in the Dockerfile. Adjust project dependencies and log levels to incorporate OpenTelemetry without disrupting existing functionality.
Updated `docker-compose.yml` to include `depends_on` for `trace-lens`, ensuring it starts after `plantuml-server` and `postgres`. Modified the README to include `Elsa.Workflows` in `OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES` and corrected formatting for `OTEL_RESOURCE_ATTRIBUTES`.
* Add database initialization script and update dependencies
Added a script to initialize the 'tracelens' database and modified the Docker setup to include this script. Refactored and improved the ProtoActorFeature class, added OpenTelemetry dependencies, and updated project settings.
* Enable OpenTelemetry integration for Proto.Actor
Added OpenTelemetry environment configuration details to the README and included the Proto.OpenTelemetry package in the project file. Updated the ProtoActorFeature to apply tracing with OpenTelemetry to WorkflowInstanceActor.
* Refactor VariablePersistenceManager to use primary constructor
This refactor simplifies the VariablePersistenceManager by moving the storageDriverManager initialization into the primary constructor. It removes the redundant field and constructor, aligning with the concise nature of modern C# syntax, and ensures consistency in accessing the storageDriverManager throughout the class.
* Remove unused Open Telemetry code from Program.cs
The code for configuring Open Telemetry was commented out but not removed, cluttering the file. This commit cleans up Program.cs by deleting these unused lines, maintaining a cleaner and more readable codebase.
* Add metrics and tracing configurations for ProtoActorFeature
Introduced methods to enable metrics and tracing in ProtoActorFeature. Removed redundant properties and updated the workflow runtime to utilize the new configurations.
* Add Directory.Build.props for shared project settings
Introduce Directory.Build.props to centralize common project settings and dependencies. Consolidate target framework, language version, and package references to reduce duplication. Remove redundant property definitions from Elsa.Server.Web.csproj.
* Move apps from bundles to apps folder and Elsa module to modules folder
* Move DynamicActivity.cs to Activities directory
The DynamicActivity.cs file has been moved from the Models directory to the Activities directory. This reorganization aims to ensure that the file's location correctly reflects its namespace.
* Add GetOutput method in ActivityExtensions
A new GetOutput method has been added to the ActivityExtensions.cs file. This method allows the retrieval of output with a specific name from an activity. Useful for handling complex types in workflow activities.
* Add feature check and refactor dependencies in Elsa
The commit introduces a new feature check in the `Module` class and refactors the dependencies in MassTransit features. Specifically, it enables querying for a specific feature before configuring the dispatcher endpoints, increasing flexibility and control. In addition, the responsibility for creating `IEndpointChannelFormatter` has been shifted from `MassTransitWorkflowDispatcherFeature` to `MassTransitFeature`, aligning with responsibility distribution.
Fixes#5165
* Add HasFeature method to IModule interface
The IModule interface has been updated to include two methods, HasFeature<T>() and HasFeature(Type featureType). These methods are designed to check if a specific type of feature has been configured, enhancing the functionality provided by the interface.
* Add WorkflowRuntimeFeature dependency
Removed unused namespaces from WorkflowsApiFeature class and added a new dependency on WorkflowRuntimeFeature. This change enhances the code cleanliness and ensures all required dependencies are correctly linked.
* Add activity completion functionality to multiple contexts
This commit introduces multiple methods to handle activity completion across various contexts, including ActivityExecutionContext and ActivityCompletedContext. It also includes updates to bookmark serialization and the WorkflowRuntime. The resulting changes should improve handling of activity outcomes and status updates in the application flow.
* Handle null options in DefaultWorkflowRuntime
Added null-conditional operators to prevent potential NullReferenceExceptions in DefaultWorkflowRuntime. This change ensures that even if the 'options' object is null, the code will not throw an exception and will instead use default values where applicable.
* Add ElsaDbContextOptions to DbContextOptionsBuilder
A line of code is added to enable applying ElsaDbContextOptions as default in DbContextOptionsBuilder within PersistenceFeatureBase. This change specifies the use of ElsaDbContextOptions when configuring the context options, enhancing the database context setup in the EntityFrameworkCore.Common module.
* Remove whitespace in Elsa.Server.Web.csproj
This commit removes unnecessary whitespaces at the end of the ProjectReference and PackageReference elements, in the Elsa.Server.Web.csproj file. This improves the readability and alignment of the code and follows the best practice for XML file format.
* Add MongoDB to docker-compose.yml
A MongoDB service has been added to the docker-compose file. The configuration includes port mapping and volume mapping for MongoDB data storage. This allows more flexibility in our environment setup with MongoDB now being spun up automatically.
* Add collection check in MongoDbStore before bulk save
Adjusted code structure, and divided longer lines of code into smaller, multi-line chunks for better readability. This refactoring makes the underlying operations and structuring of the code more apparent, aiding in future code maintenance and understanding.
* Change target branch in packages.yml workflow
This commit modifies the Github actions workflow for packaging. The branch from which to fetch changes is now specified explicitly as 'origin/patch/3.1.1' instead of the default 'origin/main'. This adjustment is specific for package creation under certain conditions.
Create new docker-compose files for non-data dog and data dog services. New services include Postgres, cockroachdb, rabbitmq, redis, and elsa-server. Along with the services, a new bash script for migration and various new activities and endpoints are also added for the Elsa server. This set of changes broadens the operational environment of Elsa.
