Commit graph

404 commits

Author SHA1 Message Date
Sipke Schoorstra 7b75f0c89f
Implement retry attempt capturing (#6674)
* Update default initializations and input parameters to `null`

Replaced `default!` with explicit `null` for input parameters and properties throughout various classes. Adjusted constructors' default values for consistency and readability. This change ensures better clarity and alignment with nullable reference types.

* Add Polly-based resilience integration for retry tracking

Introduce Polly diagnostics to log retry events in the execution context. Updated resilience strategy interfaces and implementations to support Polly's context and retry event tracking.

* Refactor resilience and retry handling, add flaky endpoint.

Removed custom Polly-based diagnostic listeners and observers in favor of a transient status code utility class. Introduced a mock "flaky" endpoint for testing failure scenarios and updated configuration for resilience strategies. Minor namespace fixes

* Add retry attempt recording to resilience feature

Introduce `IRetryAttemptRecorder` and its implementations to enable recording of retry attempts during activity execution. Updated `ResilientActivityInvoker` to persist retry attempts and modified `ResilienceFeature` to support configurable retry attempt recorders.

* Add retry attempt tracking and retrieval functionality

Introduced mechanisms to track and fetch retry attempts, including new interfaces, reader implementations, API endpoints, and related models. These enhancements improve resilience tracking and data access for activity execution across workflows.

* Add GetOutcome method to RetryAttempt model

Introduce a GetOutcome method to encapsulate logic for determining the retry attempt's outcome. It prioritizes the Result, falls back to the Exception message, or defaults to "Unknown" if neither is available. This improves clarity and reusability of the outcome evaluation.

* Add scoped registration for _retryAttemptReader

This change ensures that _retryAttemptReader is registered in the DI container as a scoped service.

* Refactor retry mechanism to support detailed retry metadata

Introduced a `CollectRetryDetails` method to `IResilientActivity` for enhanced retry data collection. Updated `RetryAttemptRecord` to include a `Details` dictionary for capturing metadata, replacing previous `Result` and `Exception` fields. These changes simplify the retry recording process and improve extensibility for tracking retry details across activities.

* Add support for capturing background activity properties

Introduced functionality to capture and persist background activity properties during workflow execution. This includes defining a key for properties, capturing them in middleware, and storing them in the workflow execution context. These changes ensure properties are handled consistently alongside other activity data.

* Add support for storing and propagating activity execution properties

Introduced a `Properties` dictionary to track additional metadata in activity execution records and stats, enabling richer diagnostics and tracing. Refactored resilience logic to improve retry handling and propagate retry-related flags in workflows. Enhanced database queries to map serialized properties for execution summaries.

* Add retry propagation for background activity execution

Introduced a mechanism to propagate the retry-attempted flag across activity execution contexts. Added a new notification `BackgroundActivityExecutionCompleted` and updated related middleware to send this notification. Enhanced resilience features to handle and propagate retry state effectively.

* Refactor default parameters and values to use 'null'.

Replaced 'default' with 'null' for optional parameters and values in `AddExecutionLogEntry`, improving clarity and ensuring semantic consistency with nullable types. No functional changes were introduced.

* Refactor flaky endpoint and enhance resilience support.

Replaced the "Flaky" endpoint with a more robust "SimulateResponseEndpoint" under a new module. Introduced a status code lookup utility and improved resilience strategies with configurable backoff types. Updated serialization to support enum conversions and enhanced caching behavior for response simulation.

* Update activity execution models with nullable properties

Replaced `default!` initializations with `null!` to ensure correct handling of nullable string properties in `ActivityExecutionRecord`. Added a new `Properties` dictionary to `ActivityExecutionRecordSummary` to store additional activity execution data. This enhances model flexibility and data extensibility.

* Add support for recording resilience strategy in context

Introduced a new method to store resilience strategy details in the activity execution context for enhanced diagnostics. Updated `ResilientActivityInvoker` to serialize and set the resilience strategy using this method, leveraging `JsonSerializer`.

* Remove redundant PropertyNamingPolicy assignment

The PropertyNamingPolicy was set to the default value (CamelCase), making the assignment unnecessary. This change simplifies the code while maintaining existing functionality.

* Set JSON property naming policy to camelCase

Updated JSON serialization settings to use camelCase naming for property names. This improves consistency with standard JSON naming conventions and ensures compatibility with camelCase-based APIs.

* Remove unused Endpoints folder reference from project file

The Endpoints folder reference in the project file was unnecessary and has been removed. This cleanup helps maintain a tidy and accurate project structure.

* Remove unused RetryAttemptFilter and add Polly packages

Removed the obsolete RetryAttemptFilter class as it was no longer in use. Added Polly and Polly.Extensions packages to the project to support resilience and fault-handling strategies. This update aligns with keeping dependencies relevant and reducing unused artifacts.

* Add resilience integration test for FlowSendHttpRequest (#6692)

* Refactor and fix resilience test cases for clarity and accuracy

Simplified imports, adjusted code structure, and corrected attempt indexing logic in resilience tests. These changes improve readability, maintainability, and ensure accurate validation of retry attempts in test scenarios.
2025-05-26 11:47:09 +02:00
Sipke Schoorstra 9aa239719d
Add IResilienceStrategy Abstraction with Category Matching and Expression-Based Configuration (#6637)
* Add resilience module with core interfaces and services

Introduced a new `Elsa.Resilience` module and its core components to support resilient services and activities. This includes resilience strategies, providers, and attributes, along with integration into the existing HTTP module for enhanced fault tolerance. Added solution and project references for proper dependency management.

* Add resilience strategy framework with HTTP strategy support

Introduced a resilience strategy architecture, including a configurable `HttpResilienceStrategy` with retry capabilities, strategy serialization, and integration with existing modules. Enhanced ResilienceFeature to support registration of strategy types and updated application configuration to enable resilience strategies. This change ensures more robust and fault-tolerant HTTP request handling.

* Add JSON serialization support for resilience configuration

Introduced `ConfigurationExtensions` to enable JSON serialization of configuration sections. Updated resilience strategies to utilize the new extension methods and adjusted JSON serialization logic to support polymorphism with `$type` discriminator. Minor modifications were made to support deserialization and property mutability.

* Add resilience strategy support to workflows and API clients

Introduced resilience strategy configuration, serialization, and execution support across workflows and API clients. Added new APIs, models, and services to enhance fault tolerance capabilities for activities and HTTP interactions.

* Refactor resilience services for improved modularity.

Replaced `IResilienceService` with new modular interfaces (`IResilienceStrategyCatalog`, `IResilienceStrategyConfigEvaluator`, `IResilientActivityInvoker`) and corresponding implementations. Enhanced maintainability by simplifying components and responsibilities, ensuring better separation of concerns.

* Rename methods in ResilienceStrategyCatalog for clarity

Updated method names in `ResilienceStrategyCatalog` and its interfaces for better readability and alignment with naming conventions. Replaced `GetAllStrategiesAsync` with `ListAsync` and `GetStrategyAsync` with `GetAsync` across the codebase.

* Refactor resilience handling in HTTP activities.

Replaced `ResilienceCategory` property with `ResilienceCategoryAttribute` for a cleaner implementation. Updated `IResilientActivity` to simplify its interface and adjusted related modifications accordingly. Introduced `IResilientActivityInvoker` to enhance resilience strategy execution.

* Add support for additional resilience and scripting features

Extended resilience strategy handling with serialization support, added `HttpResilienceStrategy` type in JavaScript handler, and refined object conversion logic for interfaces. Minor adjustments to `Expression` class properties for consistency.

* Remove `UseResilience` call from Program.cs

This call was redundant and no longer necessary for the application. Its removal simplifies the code and ensures only required middleware is used.

* Remove commented-out JSON converter code in serializer setup

Cleaned up unused and commented-out converter initialization code in `ResilienceStrategySerializer`. This improves readability and removes unnecessary clutter from the file.

* Fix typo in method names from 'Resiliency' to 'Resilience'

Renamed methods to maintain consistency in naming conventions across the codebase. This change ensures clarity and alignment with established terminology.

* Refactor namespace for ConfigurationResilienceStrategySource

Updated the namespace of ConfigurationResilienceStrategySource to "StrategySources" for better alignment with naming conventions and structure. Removed an unused namespace reference in ResilienceFeature for cleanup.

* Mark EnableResiliency as obsolete in SendHttpRequestBase.

The EnableResiliency property is now marked with the [Obsolete] attribute. Developers are encouraged to use the common Resilience Strategy setting instead for managing HTTP request resiliency. This change ensures better consistency and alignment with the broader resilience strategy.

* Restrict ResilienceCategoryAttribute to class targets only

Removed support for using ResilienceCategoryAttribute on properties. This change enforces a stricter and more focused usage of the attribute, ensuring it applies only to class-level declarations.

* Add documentation for IResilientActivityInvoker interface

Include summaries and parameter descriptions for the `InvokeAsync` method. This improves code clarity and helps developers understand the functionality and usage of the resilient activity invocation process.

* Fix logical operator precedence in type comparison check

Parentheses were added to ensure correct evaluation of conditions when checking type compatibility. This prevents potential logical errors when determining the target type in object conversions.

* Add support for resilience source identification

Introduce the `ResilienceSourceNameAttribute` to allow naming of resilience sources. Updated `ResilienceStrategyCatalog` to utilize the attribute for prefixing strategy IDs, improving source identification and traceability. Applied the attribute to `ConfigurationResilienceStrategySource` as an example.

* Revert "Add support for resilience source identification"

This reverts commit 19b4e7121d6330b5de4f692b78da4c1e4a2d1f67.

* Reapply "Add support for resilience source identification"

This reverts commit 8bcba9d040c4eb247077aec6d90dc02817adcbd5.

* Revert "Reapply "Add support for resilience source identification""

This reverts commit ee04d35e7930956c752dda3ed150ca34a535e66c.
2025-05-12 10:09:04 +02:00
Sipke Schoorstra 4266421a4c
Refactor Fault Propagation and Simplify Bookmark Management (#6545)
* Refactor bookmark management and add new features

Streamlined bookmark handling by eliminating temporary storage in `ActivityExecutionContext` and directly managing bookmarks in `WorkflowExecutionContext`. Documented architectural decisions using ADRs.

* Regenerate EF Core migrations

* Refactor fault tracking to use AggregatedFaultCount property.

Replaces FaultCount with AggregatedFaultCount across the codebase to improve clarity and consistency in fault tracking. Updates related methods, properties, and data mappings to align with the new terminology. Fixes initialization issue with nullable inputs in Fault class.

* Add migration to track fault counts in runtime (V3.5)

This migration adds a new column, "AggregatedFaultCount," to the "ActivityExecutionRecords" table. The column is an integer, non-nullable, with a default value of 0, and enables tracking aggregated fault occurrences. The migration also includes a rollback to remove this column if needed.

* Fix typo in ADR 0004 regarding bookmark management convention

Corrected a spelling mistake in the ADR documentation by changing "determins" to "determines." This ensures clarity and maintains the professionalism of the document. No functional changes were made.

* Refine fault propagation logic for child-parent activities

Replace automatic fault transitions of parent activities with an aggregate fault count for descendant activities. This avoids premature state changes while still indicating child activity faults, improving workflow resilience and accuracy.

* Remove signal-driven fault propagation ADR and renumber bookmarks ADR

The ADR for signal-driven fault propagation was deleted, and the direct bookmark management ADR was renamed and renumbered accordingly. Related references in the table of contents, graph, and solution file were updated to reflect these changes.

* Refactor DeleteBookmarks to improve readability.

Reformatted the BookmarkFilter initialization for better clarity and maintainability. This change ensures the code is more aligned with modern C# conventions and improves overall readability. No behavior or functionality has been altered.

* Refactor naming for "AggregatedFaultCount" to "AggregateFaultCount"

Standardized the terminology across the codebase and migrations by renaming all references of "AggregatedFaultCount" to "AggregateFaultCount" for improved consistency and readability. Updated relevant logic, models, migrations, and database contexts accordingly.
2025-04-10 15:01:49 +02:00
Sipke Schoorstra 51dff1a061
Support for Writing Custom Trigger Activities Using Existing Trigger Infrastructure (#6527)
* Refactor bookmark naming to use "Name" instead of "ActivityTypeName".

Replaces usages of "ActivityTypeName" with "Name" across relevant classes, filters, and database mappings for clarity and consistency. Maintains backward compatibility where necessary and updates corresponding indices, filters, and methods for proper functionality.

* Add migration for V3_5 with schema updates for EF Core

This migration modifies the `Triggers` and `Bookmarks` tables by adding the `Name` column, updating its nullable state, and creating corresponding indexes. Changes apply to both SQLite and MySQL contexts, ensuring compatibility across databases.

* Refactor bookmark filtering and tenant events handling.

Unified the bookmark filtering logic with overloads that accept multiple names, improving flexibility and reusability. Simplified object initializations in tenant events to enhance code readability and reduce verbosity. Added task continuation logic to background task execution for better task sequencing and error handling.

* Refactor bookmark creation to use target-typed `new` expressions.

Replaces explicit `CreateBookmarkArgs` instantiations with concise target-typed `new` expressions for improved readability and reduced redundancy. This does not alter functionality but simplifies the code structure.

* Make callback parameters optional and adjust OTEL settings

Updated methods to allow optional callbacks for improved flexibility. Refactored Delay activity to reuse helper methods. Adjusted OTEL instrumentation settings to enable console exporter and disable manual instrumentation.

* Refactor stimulus handling and streamline event workflows.

Introduces `WaitForEvent` and `GetEventInput` extensions to improve ActivityExecutionContext usability. Replaces generic filter methods with targeted single-name filtering, simplifying scheduling logic. Updates stimulus names for consistency and deprecates obsolete methods to enhance clarity and maintainability.

* Refactor `Event` activity handling and event stimulus logic.

Replaces inline event stimulus creation with a new `GetEventStimulus` helper method for cleaner code. Simplifies event execution handling by removing redundant logic in `ExecuteAsync`. Extends `WaitForEvent` to handle workflow triggers more efficiently.

* Refactor and enhance Timer and Delay execution logic

Introduced `TimerBase` for shared timer functionality and refactored `Timer` to extend it. Improved method names for clarity, replacing `ResumeIn`/`ResumeAt` with `DelayFor`/`DelayUntil`. Enhanced flexibility in bookmark handling and activity execution context extensions.

* Add custom activities and refactor HTTP stimulus handling

Introduce new custom activities (CustomDelay, CustomEvent, CustomHttpEndpoint, CustomTimer) to enhance workflow functionality. Refactor HTTP stimulus handling by replacing activity type names with a new centralized HttpStimulusNames constant, improving consistency and maintainability. Additionally, streamline HTTP endpoint logic with new helper extensions and simplify related services to reduce redundancy.

* Add HttpEndpointBase abstraction to simplify HTTP endpoints

Introduce a new `HttpEndpointBase` class to centralize common logic for HTTP endpoint activities. Refactored `CustomHttpEndpoint` to inherit from this new base class, reducing redundancy and improving maintainability.

* Refactor events framework with base class for event activities

Introduce `EventBase` to streamline implementations of event-driven activities. Updated `CustomEvent` to inherit from `EventBase`, reducing duplicate logic and improving maintainability. Removed unnecessary dependencies in `CustomTimer`.

* Refactor bookmark creation to use target-typed `new` expressions.

Replaces explicit `CreateBookmarkArgs` instantiations with concise target-typed `new` expressions for improved readability and reduced redundancy. This does not alter functionality but simplifies the code structure.

Make callback parameters optional and adjust OTEL settings

Updated methods to allow optional callbacks for improved flexibility. Refactored Delay activity to reuse helper methods. Adjusted OTEL instrumentation settings to enable console exporter and disable manual instrumentation.

Refactor stimulus handling and streamline event workflows.

Introduces `WaitForEvent` and `GetEventInput` extensions to improve ActivityExecutionContext usability. Replaces generic filter methods with targeted single-name filtering, simplifying scheduling logic. Updates stimulus names for consistency and deprecates obsolete methods to enhance clarity and maintainability.

Refactor `Event` activity handling and event stimulus logic.

Replaces inline event stimulus creation with a new `GetEventStimulus` helper method for cleaner code. Simplifies event execution handling by removing redundant logic in `ExecuteAsync`. Extends `WaitForEvent` to handle workflow triggers more efficiently.

Refactor and enhance Timer and Delay execution logic

Introduced `TimerBase` for shared timer functionality and refactored `Timer` to extend it. Improved method names for clarity, replacing `ResumeIn`/`ResumeAt` with `DelayFor`/`DelayUntil`. Enhanced flexibility in bookmark handling and activity execution context extensions.

Add custom activities and refactor HTTP stimulus handling

Introduce new custom activities (CustomDelay, CustomEvent, CustomHttpEndpoint, CustomTimer) to enhance workflow functionality. Refactor HTTP stimulus handling by replacing activity type names with a new centralized HttpStimulusNames constant, improving consistency and maintainability. Additionally, streamline HTTP endpoint logic with new helper extensions and simplify related services to reduce redundancy.

Add HttpEndpointBase abstraction to simplify HTTP endpoints

Introduce a new `HttpEndpointBase` class to centralize common logic for HTTP endpoint activities. Refactored `CustomHttpEndpoint` to inherit from this new base class, reducing redundancy and improving maintainability.

Refactor events framework with base class for event activities

Introduce `EventBase` to streamline implementations of event-driven activities. Updated `CustomEvent` to inherit from `EventBase`, reducing duplicate logic and improving maintainability. Removed unnecessary dependencies in `CustomTimer`.

* Move HttpEndpointOptions model to its own file

The HttpEndpointOptions class was moved from an extension file to its own dedicated file for better organization and modularity. This model defines HTTP endpoint properties such as path, methods, authorization, policies, request timeout, and size limit. The change improves code clarity and structure.

* Fix unnecessary whitespace in Timer.cs

Removed an extra whitespace line in the Timer.cs file to maintain code formatting consistency. No functional changes were made to the code.

* Remove extraneous whitespace in IStimulusSender.cs file

Eliminate unnecessary blank line in the IStimulusSender interface for improved code cleanliness. This change enhances readability and aligns with coding standards.
2025-03-21 23:16:56 +01:00
Sipke Schoorstra b543366a4b
Refactor trigger indexing and update base version to 3.5.0. (#6516)
Refactored `TriggerIndexingContext` to enhance immutability and streamline trigger handling logic. Added support for generating trigger names dynamically and improved bookmark creation in HTTP workflows. Removed unused Docker Compose file and updated the base version in the build workflow to `3.5.0`.

Fixes #6512
2025-03-20 12:04:51 +01:00
Sipke Schoorstra fd31b5b605
Update OTEL fields and tags
Added and updated multiple package references, including OpenTelemetry, Datadog.Trace.Bundle, and various Microsoft.Extensions libraries. This ensures compatibility with the latest dependencies and introduces enhanced features for resilience and tracing.
2025-03-14 23:17:54 +01:00
Sipke Schoorstra 5be82dd9ab
Add support for Citus and YugabyteDB integration
Introduced Docker Compose configurations for Citus and YugabyteDB clusters and updated connection strings in `appsettings.json`. Modified `Program.cs` to enable Entity Framework Core support for both databases and updated the `SqlDatabaseProvider` enum accordingly. Adjusted the Docker Compose setup and bumped `Yarp.ReverseProxy` package version.
2025-03-11 11:19:59 +01:00
Sipke Schoorstra 74a1c50f66
Remove Agents projects
Deleted Agents projects. These projects are being migrated to elsa-integrations.
2025-03-02 15:55:29 +01:00
Sipke Schoorstra 42ab1a8944
Merge branch 'main' into feat/4832 2025-03-02 14:50:21 +01:00
Sipke Schoorstra 43157fb6d6
Remove Azure Service Bus integration and related tests
Deleted the entire Azure Service Bus module, including activities, configurations, tests, and supporting code. This removal eliminates features like message handling workflows, queue/topic providers, and unit testing utilities for Azure Service Bus.
2025-02-23 23:43:15 +01:00
Sipke Schoorstra ce0ea7e680
Update package and remove obsolete project entry
Upgraded `System.Formats.Asn1` to version `9.0.2` to address vulnerabilities. Removed the obsolete "nuke" project entry and associated files from the solution to clean up unused references. These changes enhance security and streamline the project structure.
2025-02-23 23:38:48 +01:00
Sipke Schoorstra 7aabfb6d0b
Remove Elsa.OrchardCore module.
Deleted the entire Elsa.OrchardCore module, including all its associated activities, handlers, services, and extensions. This removes support for integrations with Orchard Core application instances.
2025-02-23 20:10:16 +01:00
Sipke Schoorstra b9e9459e94
Remove Telnyx module from the codebase
Eliminated all source files related to the Telnyx module, including activities, payloads, extensions, and models. This effectively deprecates Telnyx-related functionalities from the system in favor of the new Elsa.Integrations.Telnyx module in the [elsa-integrations](https://github.com/elsa-workflows/elsa-integrations) repo.
2025-02-23 14:47:19 +01:00
Sipke Schoorstra e2ec2e47b4
Add Connections modules and update dependencies.
Added new Connections modules to the solution and updated project references to include necessary dependencies. Also modified DictionaryExtensions to return null as the default value instead of default. Removed an unused JavaScript helper import to streamline the code.
2025-02-23 13:46:59 +01:00
Sipke Schoorstra dd92fa097c
Add migration files for database schema V3.4
This commit introduces migration files for multiple database providers (MySQL, SQL Server, PostgreSQL, SQLite, and Oracle) to support schema changes for version 3.4. The migrations primarily involve mapping entity schemas, indexes, and table structures for consistency across all supported database systems.
2025-02-22 20:48:56 +01:00
Sipke Schoorstra 0a21b5201a
Add workflow restart functionality for handling interruptions
Introduced a mechanism to identify and restart interrupted workflows. This includes a new `IWorkflowRestarter` contract, its default implementation, and a recurring task for handling restarts. Additionally, updated configurations and added extensions to improve workflow instance filtering and liveness tracking.
2025-02-22 15:10:52 +01:00
Sipke Schoorstra 689fd6fecf Remove duplicate Kafka compose entry and add oracle setup.
Duplicate reference to `docker-compose-kafka.yml` was removed to tidy up the solution file. Added
2025-02-09 08:36:39 +01:00
Matt 1029580712
Merge branch 'main' into main 2025-02-06 21:20:12 +00:00
Sipke Schoorstra 6c0d1ea66c Refactor database setup and EFCore provider configurations.
Revised database initialization scripts to better handle Postgres and Oracle environments, introduced schema-specific configurations for Oracle EFCore, and cleaned up obsolete or redundant entity model setup. Streamlined project structure by relocating and renaming files for SQLite and Oracle EFCore setups, improving maintainability and readability.
2025-01-31 19:51:24 +01:00
Matthew Knibbs c4e04c4434 Adds SQL Activities. Includes implimentations for MS SQL Server, PostgreSql, MySql and Sqlite. 2025-01-01 23:14:09 +00:00
Matthew Knibbs 97a0b36122 Correct JsonEditor namesapce 2024-12-31 10:22:28 +00:00
Sipke Schoorstra f53e024d25
Add Elsa.Kafka Module for Kafka Integration with Message Sending and Receiving Activities (#6108)
* Add Kafka module with integration and example setup

Introduced the Kafka module providing consumer integration and activities into the project. This includes new classes for consumer handling, configuration, and activities. An example setup using Docker Compose is also added to facilitate development and testing.

* Refactor KafkaTransportMessage to inline Timestamp namespace

Simplify the namespace usage for the Timestamp type within the KafkaTransportMessage record. This change eliminates the need for a separate using directive for Timestamp, enhancing code readability and maintainability.

* Add support for handling Kafka transport messages

This commit introduces the capability to handle and trigger workflows based on Kafka transport messages. It adds a new handler, notifications, and updates the message stimulus to include correlating fields. Additionally, the Kafka consumers are now managed more modularly with updated startup tasks and mediator integration.

* Enable Kafka integration and fix Kafka options naming

Added support for Kafka integration in Elsa.Server.Web by setting up Kafka configurations in appsettings.json and updating Program.cs. Also, renamed `ConsumerConfigs` to `ConsumerDefinitions` in Kafka options for clarity.

* Add consumer definition enumeration and dropdown support

Introduced `IConsumerDefinitionEnumerator` for managing consumer definitions across providers and implemented in `ConsumerDefinitionEnumerator` class. Enhanced `KafkaFeature` to register these services and updated the `MessageReceived` activity to use a dropdown UI hint for consuming definitions. Improved `StartConsumersTask` by refactoring consumer definition retrieval logic.

* Add SendMessage activity and refine Kafka messaging

Introduce a new SendMessage activity for Kafka, enabling message publishing to specific topics. Refine KafkaTransportMessage model by removing headers and timestamp fields. Adjust the StimulusSender logic to streamline the bookmark queuing process and fix key-value pairing in dropdown options. Update appsettings for corrected Kafka bootstrap server and topic configurations.

* Add producer and topic management support

Introduced interfaces and implementations for managing producer and topic definitions along with their respective enumerators and list providers. Updated `SendMessage` activity to include producer selection and refactored consumer definition providers for better consistency.

* Refactor Kafka configuration property names

Renamed Kafka configuration properties for better consistency and readability across the codebase. Updated property names from `ProducerDefinitions` to `Producers`, `ConsumerDefinitions` to `Consumers`, and `TopicDefinitions` to `Topics`. Added missing input attribute in `SendMessage.cs` and registered additional handlers in `KafkaFeature.cs`.

* Add custom serializers for Kafka message handling

Introduced `DefaultSerializers` class for custom serialization and deserialization of Kafka messages. Updated `MessageReceived` and `SendMessage` activities to use these custom serializers, and modified `KafkaOptions` to include them.

* Fix ExpandoObject serialization method parameter

Changed the serialization type from `ExpandoObject` to the actual type of the object to ensure proper serialization. This ensures that derived types are correctly handled during the serialization process.

* Add Producer and Consumer workflows for Kafka

Introduced two new workflows: `ProducerWorkflow` and `ConsumerWorkflow` for handling Kafka messages. Updated `DefaultSerializers` to use camelCase property naming and modified `appsettings.json` to include `topic-2` and format entries.

* Add JSON serialization to log output in ConsumerWorkflow

This change enhances the log output by serializing messages to JSON format before writing them. The addition of System.Text.Json ensures that the message content is presented in a structured and standardized format in logs.

* Add correlation strategies and update Kafka features

Implemented HeaderCorrelationStrategy and NullCorrelationStrategy, and updated KafkaFeature to support customizable correlation strategies. Added correlation ID handling to Kafka transport messages and updated config and handlers accordingly.

* Add tenant accessor to ConsumerDefinitionWorkflowContextProvider

Integrated ITenantAccessor to the provider to support tenant-specific context loading. Updated the constructor and LoadAsync method to retrieve the tenant information and use it for context-specific operations.

* Switch to MySQL and disable Kafka

This commit changes the SQL database provider from SQLite to MySQL and disables Kafka use. It also includes necessary adjustments such as adding MySQL handling in configuration and connection setups, updating `docker-compose` to include MySQL services, and referencing MySQL projects in the `.csproj` file.

* Add UI property handlers to multiple features

This commit introduces various UI property handlers across several features such as Python, JavaScript, CSharp, and Workflow features to enhance user interface property handling. It also updates the property UI handler resolution logic to better manage cases where providers are not available. Furthermore, adjustments were made in the server configuration to switch database providers and enable Kafka.

* Refactor property UI handler retrieval logic

Modified the logic to fetch property UI handlers by preloading them into a list and then filtering. This change improves readability and potentially performance by reducing repetitive service provider calls.

* Incremental work on Kafka workers and predicate evaluation

* Merge BookmarkInvoker with BookmarkResumer

* Register IWorkerManager

* Change lifetime scope of WorkerManager to Singleton

* **Introduce topic subscription handling for Kafka workers**

Added `IWorkerTopicSubscriber` interface and its implementation for managing topic subscriptions. Enhanced workers to bind triggers and bookmarks dynamically based on existing data. Updated several classes and methods to support topic-based subscriptions and headers.

* Refactor trigger matching logic.

Extract trigger matching conditions into `IsMatchAsync` method for reuse. This enhances code maintainability and readability by reducing redundancy. The new `GetTopic` helper method isolates the topic retrieval logic.

* Add Name property to MassTransitActivityTypeProvider

This commit inserts the Name property in the returned object within the MassTransitActivityTypeProvider class. It ensures that the typeName is included, providing a clearer definition of the activity type.

* Add handling for deleted bookmarks and refactor bookmark removal

Added a new event handler for `BookmarksDeleted` to ensure removed bookmarks are processed correctly. Refactored the bookmark removal logic into a helper method to reduce code duplication and streamline the workflow.

* Switch to asynchronous bookmark queue processing

Refactored the `TriggerWorkflows` handler to use `IBookmarkQueue` instead of directly invoking the `IBookmarkResumer`. This change aims to improve scalability by queueing bookmark resumption requests, enabling better load distribution and async processing. Added necessary helpers and configuration options to support this functionality.

* Remove unused IBookmarkResumer dependency

Simplify the constructor by removing the unused IBookmarkResumer dependency. This cleanup reduces potential confusion and improves code maintainability without impacting functionality.

* Add support for local message processing

Introduced an `IsLocal` property to `MessageReceivedStimulus` for determining if the message event is local to a specific workflow instance. Updated `BookmarkBinding` and related handler methods to utilize `CorrelationId` for local event matching. Removed unused `CorrelatingFields` from `MessageReceived` activity.

* Add nullability checks to IWorker retrieval methods

Updated `GetWorker` methods to return nullable `IWorker` to handle cases where a worker might not exist. Modified code to include null checks and conditional operations to prevent potential null reference exceptions when accessing worker methods.

* Add filtering based on activity type name for triggers and bookmarks

This commit introduces filtering for triggers and bookmarks based on the `MessageReceived` activity type name. It also adds an option to mark messages as local in the `SendMessage` activity, where local messages are delivered to the current workflow instance only. These changes help enhance the management and targeted delivery of messages within the workflow framework.

* Add new Kafka topics and clean up producers config

New topics "topic-3" and "topic-4" were added to the Kafka settings. Unused topic references were removed from the producers configuration to simplify and improve clarity.

* Add predicate to KafkaConsumerActivity in ConsumerWorkflow

Introduced a predicate to the KafkaConsumerActivity using JavaScript expressions to filter messages based on OrderId. This ensures only relevant messages are processed in the workflow.
2024-11-18 13:42:54 +01:00
Sipke Schoorstra 50e23ba738 Add workflow to close stale issues and PRs
Introduced a new GitHub Actions workflow to automatically mark issues and PRs as stale and close them after a period of inactivity. This helps in managing the project's issue tracker by ensuring old, inactive issues are addressed or closed.
2024-11-12 11:19:09 +01:00
Sipke Schoorstra 98d914f29d Remove Hangfire integration sample project
Removed the entire Hangfire integration sample project, including all associated files and configurations. This cleanup ensures the solution is leaner and easier to maintain, focusing on core components.
2024-11-04 22:55:19 +01:00
Sipke Schoorstra a397b6799e Add new sample project for ASP.NET Hangfire Integration
Introduced a new sample project to demonstrate ASP.NET integration with Hangfire, including setup configurations in `appsettings.json`, program initialization, and a background job activity. This will serve as a reference implementation for integrating Elsa with Hangfire in ASP.NET applications.
2024-11-04 11:59:47 +01:00
Sipke Schoorstra 6be9e53de1 Add new project reference to solution file
This commit updates Elsa.sln to include a new project reference with GUID {2CDF3E1C-267D-4198-B1C7-7E1F548FC120}. This ensures that the project dependencies are correctly tracked within the solution.
2024-10-30 11:53:57 +01:00
Sverre Winkelmans d899b674dc
Retention Module (#5344)
* Retention module

* Remove unused batch size

* Use ISystemClock instead of DateTime.Now

* Remove TargetFramework

* Use WorkflowInstanceManager

* Update using statement, remove cloneable from WorkflowInstanceFilter.cs

* Update project description and remove unnecessary properties

* Move to BackgroundService

* Generic retention module

---------

Co-authored-by: Sipke Schoorstra <sipkeschoorstra@outlook.com>
2024-10-28 19:44:23 +01:00
Sipke Schoorstra 0c10113d7b Remove outdated database migration files
Deleted obsolete migration files from SQL Server, MySQL, and SQLite projects. This cleanup removes unused schema definitions and indexes, ensuring the repository remains up to date and free of unnecessary files.
2024-10-23 14:32:50 +02:00
Sipke Schoorstra 7e7a899bbf
Implement multitenant HTTP routing (#6031)
* Add tenant awareness to bookmark handling and route resolution

Added tenant ID support across various components, including bookmark updates, route resolution, and middleware processing. This ensures that bookmark and route operations can now appropriately handle tenant-specific data, improving the system's multitenancy capabilities.

* Add Multitenant HTTP Routing feature to Tenants module

Introduced a new MultitenantHttpRoutingFeature class to the Elsa.Tenants.AspNetCore module, enhancing the tenant resolution capabilities. Moved RoutePrefixTenantResolver from Elsa.Http to Elsa.Tenants.AspNetCore and updated relevant project references and namespaces accordingly. This refactor improves modularity and separation of concerns between HTTP and tenancy features.

* Refactor route handling and tenant configuration

Removed redundant `RouteTableExtensions` and replaced with new route providers and updaters, enhancing flexibility and modularity. Introduced tenant-specific HTTP endpoint configurations for better customization and configuration management.

* Rename HttpEndpointBookmarkStimulus to HttpEndpointBookmarkPayload

Refactor various classes and methods to reflect the renaming from `HttpEndpointBookmarkStimulus` to `HttpEndpointBookmarkPayload`. Add and configure new extension methods for tenant route handling, update the route provider to support multi-tenancy, and adjust the tenants provider to bind configuration properly.

* Add HeaderTenantResolver and refactor Http namespace.

Introduce HeaderTenantResolver to resolve tenants via HTTP headers. Refactor multiple classes and interfaces to move from the Elsa.Http.Models namespace directly into Elsa.Http for clarity and consistency.

* Add Host-based tenant resolution

Implemented a HostTenantResolver to resolve tenants based on the request's host and updated tenant configurations with host information. Modified the tenant resolver pipeline and added the new host resolver to the service registrations.

* Add tenant-aware caching and accessor support

Enhanced caching by incorporating tenant identifiers into cache keys for more granular cache management. Introduced ITenantAccessor dependencies in various services to retrieve the current tenant information. This ensures that cache entries are correctly isolated per tenant.

* Reorder tenant resolvers for pipeline setup.

Reordered the tenant resolvers in the pipeline to prioritize HostTenantResolver before RoutePrefixTenantResolver. This ensures that tenant resolution is correctly aligned with host-based resolving before checking the route prefix.

* Remove unused imports

This commit eliminates redundant `using` directives across multiple files to streamline the codebase. This cleanup helps improve code readability and maintainability by removing unnecessary dependencies.
2024-10-14 21:27:11 +02:00
jdevillard 9808020d57
Add db providers for secrets features (#6020)
* Add PostgreSql provider to Secrets

* Add SqlServer provider to Secrets

* fix sqlserver Method name

* resolve conflict Elsa.sln

* resolve conflict Elsa.sln

---------

Co-authored-by: Jérémie DEVILLARD <jdevillard@users.noreply.github.com>
2024-10-14 21:03:39 +02:00
Sipke Schoorstra 121bc52b07 Remove sample files.
Samples are moved to the elsa-samples repository.
2024-10-12 19:19:10 +02:00
Sipke Schoorstra a5cc3fc9e8
Refactor Tenant Resolution to Use Async Local Storage for Operation-wide Access (#6022)
* Remove obsolete tenant-related classes and add ASP.NET Core middleware

Refactored tenant resolution by removing obsolete interfaces and classes, such as `IAmbientTenantAccessor` and `ITenantResolutionStrategy`. Introduced new ASP.NET Core middleware for tenant resolution, encapsulated in the new `Elsa.Tenants.AspNetCore` project. Updated related usage in various parts of the application to align with these changes.

* Remove HttpContextTenantResolver.

Removed HttpContextTenantResolver from the multitenancy pipeline and related service registrations. This simplifies the tenant resolution by relying on remaining resolvers like ClaimsTenantResolver and RoutePrefixTenantResolver.

* Add Elsa solution definition file

This commit introduces the main solution file, Elsa.slnx, defining the folder structure, projects, and configuration for the Elsa repository. This includes folders for Docker, documentation, pipelines, samples, scripts, source code, and tests.

* Refactor DefaultAccessTokenIssuer for clarity and efficiency

Refactored the DefaultAccessTokenIssuer class by simplifying its constructor and utilizing scoped variables for token options. Improved token creation logic by adding a dedicated method to configure token options, enhancing code readability and maintainability.

* Remove Elsa.slnx solution file

No dotnet build support yet.

* Refactor tenant resolver service registrations

Updated the service registrations to use interfaces for DefaultTenantResolver and DefaultTenantResolverPipelineInvoker. This improves the code's flexibility, making it easier to replace or extend these implementations in the future.

* Add multitenancy support and tenant scope management

Introduced ITenantScopeFactory and related implementations for tenant scope management across the application. Enhanced the HTTP workflows middleware to handle tenants and updated relevant configurations and extension methods to support tenant resolution.

* Remove unnecessary folder inclusion

The <Folder> tag for "Modules\Modules\" was redundant and has been removed to clean up the project file. This change will not affect the existing functionality or project structure.

* Rename Create to CreateScope and improve authorization.

Updated the method name from Create to CreateScope for better clarity in the TenantScopeFactory. Fixed a logical error in the authorization process, ensuring proper status code setting for unauthorized requests, and refactored token expiration calculation for clarity.

* Add tenant agnostic filters and remove tenant setup

This commit introduces tenant agnostic filters in AutoUpdateTests to ensure workflows can trigger regardless of tenant. Additionally, it removes tenant configuration from WorkflowServer setup as it is no longer required for the current tests.
2024-10-12 12:08:09 +02:00
Sipke Schoorstra f3961c0af5 Update namespaces and correct merge artefacts 2024-10-11 20:41:49 +02:00
Sipke Schoorstra ff487fad80 Merge remote-tracking branch 'origin/blueberry' into enh/multi-tenancy-refactorings 2024-10-11 19:45:40 +02:00
Sipke Schoorstra 8388f28a89 Add JavaScript library support
Introduce support for Lodash and Moment libraries in the JavaScript engine. Also configure project settings and Weavers for consistent code quality and behavior.
2024-10-06 20:29:14 +02:00
Sipke Schoorstra bbedd61138
Implement Activity State Filtering and JavaScript Integration (#5993)
* Add secret scripting integration for JavaScript

Introduced a new `Elsa.Secrets.Scripting` module that provides secret management capabilities within JavaScript workflows. This includes configuring the Jint engine to use workflow variables, adding new type and variable definition providers, and integrating with existing secret management features.

* Refactor secret name extraction to a separate method

Moved the logic for extracting secret names from the main method to a dedicated private method `GetSecretNamesFromExpression`. This improves code readability and maintains the single responsibility principle by delegating secret name extraction to its own method.

* Add input evaluation, sensitive input handling, and middleware refactor

Introduced methods for evaluating activity input properties and handling inputs marked as sensitive. Refactored `ExecutionLogMiddleware` constructor for consistency. Enhanced `SendHttpRequestBase` to mark authorization inputs as potentially containing secrets. Removed obsolete entries and adjusted persistence logic for clarity.

* Refactor IActivityStateProtector interface

Remove unused using directives and unnecessary comments. Simplify the definition of the `ProtectedActivityStateContext` record.

* Add activity state filtering mechanism

Introduce an abstract filter base class, context, and result models to enable filtering of activity state. Implement a default filter manager to run these filters and apply a specific filter for obfuscating HTTP request headers. Update necessary dependencies and extension methods to integrate the new filtering functionality.

* Add expired secrets management

Implemented services to manage expired secrets by periodically checking and updating their status. Introduced a new hosted service to perform the sweep and configurable options for the sweep interval. Updated related classes and configurations accordingly.

* Update SweepInterval in appsettings.json

Changed the Secrets Management SweepInterval from 30 seconds to 4 hours. This adjustment aims to reduce the frequency of sweep operations and improve overall system performance.

* Update comment to reflect configuring engine with secrets

The comment was changed to better describe the handler's function, specifying that it configures the Jint engine with secrets instead of workflow variables. This clarifies the purpose and usage of the handler in the context of the code.

* Remove unused inputDescriptors variable

This commit removes the inputDescriptors variable, which was declared but never used in DefaultActivityExecutionMapper.cs. This helps in cleaning up the code and potentially reducing memory usage. Ensuring that all declared variables are utilized can improve code readability and maintainability.
2024-10-02 09:11:35 +02:00
Sipke Schoorstra 758b60767b Add OpenTelemetry auto-instrumentation support
Introduce OpenTelemetry configuration and dependencies for observability. Replace DataDog tracer with OpenTelemetry auto-instrumentation in the Dockerfile. Adjust project dependencies and log levels to incorporate OpenTelemetry without disrupting existing functionality.
2024-09-25 00:49:21 +02:00
Sipke Schoorstra adf36927dc Update Docker and k8s configs, enhance OTEL tracing
Updated `docker-compose-datadog.yml` with new environment variables and updated image versions, enhancing trace sampling and service autodiscovery. Added Kubernetes deployment, service, and role files for `elsa-server`, `elsa-studio`, `plant-uml`, `postgres`, and `trace-lens`, improving the project's infrastructure. Enhanced OpenTelemetry middleware for better error handling and detailed tracing of activity and workflow executions.
2024-09-23 23:35:30 +02:00
Sipke Schoorstra 79b5bfc378 Add OpenTelemetry support for tracing workflow execution
This commit introduces the Elsa.OpenTelemetry module to provide OpenTelemetry sources for tracing workflow and activity execution. Additionally, it updates the program to include default workflow and activity execution pipelines with tracing middleware, and improves pipeline builder extensions to support middleware insertion.
2024-09-23 11:35:43 +02:00
Sipke Schoorstra 329a79be2e
Secrets API (#5967)
* Add initial implementation for Elsa Secrets modules

This commit introduces new projects for managing secrets within the Elsa framework: `Elsa.Secrets.Api`, `Elsa.Secrets.Core`, and `Elsa.Secrets.Management`. These projects include essential interfaces, models, entities, and endpoints to handle secret storage, retrieval, and management. Specific features include API endpoints for listing secrets, models for secret filtering, and interfaces for encryption key handling.

* Add weavers

* Refactor encryption handling in secrets management

Implemented a new architecture for handling encryption keys and algorithms within the secrets management system. Replaced old encryption key entities and related interfaces with a more modular and extensible approach. Added new services and models to improve encryption and decryption processes, enhancing maintainability and scalability.

* Add IEncryptor interface for encryption functionality

Introduced the IEncryptor interface to standardize encryption operations within the Elsa.Secrets.Management module. This interface includes the EncryptAsync method to handle encryption using a specified key ID and value.

* Add dependency on SecretsFeature and configure secrets provider

Integrate the SecretsFeature dependency and configure a secrets provider within the SecretsManagementFeature class. This adds the StoreSecretProvider to the service collection and ensures the secrets provider is correctly set up. Also, rename method from WithSecretsProvider to UseSecretsProvider for clarity.

* Add EF Core and SQLite support for Secrets module

Introduced Entity Framework Core and SQLite support for the Secrets module, including migration files, EF Core configurations, context factory, and store implementation. Added necessary extensions and configuration code to integrate with the existing API and features. Included updates to the main web application to utilize the new persistence providers.

* Update Microsoft.SemanticKernel to version 1.18.2

Upgrade Microsoft.SemanticKernel package to the latest version to ensure compatibility and new features. Remove unused Elsa.Agents.Persistence using directive from Program.cs for code cleanliness.

* Update workflows and add Agents module

Changed workflow branch targets from `main` to `feature/secrets`. Updated Docker image tags and added the `Agents` module to the Elsa Studio WebAssembly project.

* Enable agent activities in workflow configuration

This change introduces the `.UseAgentActivities()` method in the workflow configuration, enhancing the workflow capabilities. By doing so, it ensures that agent activities are appropriately integrated and available for use in the application.

* Add EF Core migrations for MySQL and SQL Server

Added Entity Framework Core migrations and related configurations to support MySQL and SQL Server for the Agents Persistence module. These changes include new migration files, context factories, and project configurations.

* Fix migration assembly reference and update method syntax

Changed the migration assembly reference in SqlServerProvidersExtensions. Updated method syntax in WorkflowManagementFeature to use array shorthand format.

* Add secret management functionalities

Introduced secret management services with CRUD operations, notifications, and bulk actions. Added unique name generation and validation for secrets, and implemented corresponding API endpoints.

* Enhance Secret Management Feature

Added Elsa.Extensions import and updated MemorySecretStore registration to use the AddMemoryStore method with Secret. This improves code modularity and adheres to the updated registration method conventions.

* Remove encryption services and update migration

Removed multiple files related to encryption services and their dependencies, including encryption algorithms and key providers. Also updated a migration script to reflect schema changes, removing specific columns and constraints.

* Implement versioning and retrieval for secrets management

Added "IsLatest" flag and cloning mechanism for secrets to support versioning. Introduced a new API endpoint for fetching decrypted secret input models. Refactored encryption and decryption logic to handle empty values gracefully.

* Add secret management functionalities

Introduced services and interfaces for secret name generation, validation, and updating. Updated secret handling to include expiration metadata. Refactored methods in ISecretManager to streamline secret creation and update processes.

* Remove DisableSyntaxSelection class and references

Deleted the DisableSyntaxSelection class and its references from various files. This includes removing its registration as a Scoped service and associated usage in the `RunJavaScript` activity.

* Add new migration for secrets and update DefaultSecretManager

Re-created migration files for V3_3 to include the ExpiresIn column. Updated DefaultSecretManager to utilize identityGenerator for generating Id and SecretId, and added additional fields like CreatedAt, UpdatedAt, and IsLatest.
2024-09-15 17:12:13 -07:00
Sipke Schoorstra 2d62a373e0
Add Agents Module with Semantic Kernel Integration (#5937)
* Remove `ServiceProfileConfig` and `SkillConfig`

Refactored the codebase to eliminate `ServiceProfileConfig` and `SkillConfig` in favor of simplifying configurations. Updated related entities and settings to consistently use `ServiceConfig` and `AgentConfig`. Refined function execution to enhance overall system coherence and maintainability.

* Refactor agent management and plugin system

This update eliminates AgentManager and related classes, introducing new management interfaces and memory-based stores. The plugin system is refactored with better separation of concerns and the addition of provider interfaces.

* Add new Agents features and services

Introduced AgentsFeature for API endpoints, AgentManagementFeature for agent management, and AgentActivitiesFeature for activities. Updated service configurations and migrated files for cleaner and more maintainable code structure.

* Rename execute endpoints to invoke

Renamed the 'Execute' endpoints and related paths to 'Invoke' for better semantic clarity. Adjusted the namespace and updated the route configuration accordingly to reflect this change.

* Add permissions configuration to Agents Invoke endpoint

This commit adds a call to `ConfigurePermissions("agents:invoke")` in the Agents Invoke endpoint configuration. This change ensures that proper permissions are checked when this endpoint is accessed.

* Rename Management to Persistence with EF Core support

Renamed various files and classes from Management to Persistence. Introduced Entity Framework Core (EF Core) support for agents, including implementation of APIs for API key, service, and agent definitions. The changes also updated project configurations and added necessary database context and entity configurations.

* Add EntityFrameworkCore project for Agents persistence

Introduce a new Entity Framework Core provider to manage the persistence layer of the Agents module. The project includes references to the base persistence and common Entity Framework projects.

* Remove redundant exception handlers and add new SQLite migration.

Deleted unnecessary `DbExceptionTransformer` and `RethrowDbExceptionHandler` classes to streamline error handling. Introduced a new SQLite migration for the Agents module to create necessary database tables and indices. Also modified `Store` and `EntityStore` to utilize a service provider for exception handling.

* Add Sqlite configuration extension and sample updates

Introduce a new extension method for configuring EF Core to use Sqlite in the Elsa.Agents module. Updated the sample project to use a configurable Sqlite connection string and adjusted the appsettings.json accordingly. Also added necessary folder structure in the project file.

* Add agent, service, and API key CRUD operations

Introduced CRUD functionalities for agent, service, and API key entities. Added endpoint handlers, filtering capabilities, and memory/EF core stores to support these operations. Updated configuration mappings as well.

* Add Agent API endpoints and DTOs

Added CRUD endpoints for agents in Elsa.Agents.Api along with the required DTOs and Request models. These endpoints include create, read, update, delete, and list functionalities, facilitating the management of agent entities.

* Add support for agent service providers and discovery

Introduce interfaces and implementations for agent service providers and service discovery. Updated `KernelFactory` to integrate the new service discovery mechanism and restructured plugin discoverer naming for consistency.

* Add agent management features

Introduce agent management components, including pages for listing and handling agent workflows, dialogs for creating agents with Blazor components, and backend API endpoints for generating unique agent names. This enhances functionality for managing agent workflows within the application.

* Refactor exception handling and simplify event handling logic

Replaced DataProcessingDbExceptionHandler with DbExceptionTransformer for handling PostgreSQL exceptions. Streamlined event completion logic by renaming method and ensuring input retrieval before setting the result.

* Add bulk delete endpoint for agents

Introduced new API endpoint for bulk deleting agents. Implemented request and response models, and updated persistence layer to handle bulk operations. Also reorganized project structure for better modularity.

* Refactor API Key models and endpoint classes

Consolidate API Key request models into unified models and update endpoints to use these new models. Add new transformation methods to entity classes. Enhance configuration classes and streamline the dependency injection process.

* Refactor and streamline bulk delete and service APIs

Refactored the bulk delete endpoints for API keys, services, and agents, consolidating request and response models. Added new service input and model classes and updated the corresponding stores to support bulk deletions. Simplified the endpoint logic to use these new models and enhanced filtering capabilities.

* Set default values for AgentInputModel properties

Updated properties in `AgentInputModel` to use initialized default values instead of `default!`. This change ensures that string and object properties are instantiated properly, improving code stability and preventing potential null reference exceptions.

* Add plugin listing endpoint and related models

Introduced a new endpoint to list all registered plugins in the system. Added necessary models and methods for plugin description and retrieval, along with appropriate documentation comments.

* Refactor agent management to use `IAgentManager`.

Created new notifications for agent lifecycle events and introduced an `AgentManager` service for agent CRUD operations. Updated APIs and other components to use the new `IAgentManager` service instead of direct store access for a more cohesive architecture.

* Update agent activity descriptor names

Changed the activity descriptor names to use PascalCase for consistency and readability. Also added display names for input variables to improve user interfaces.

* Add overloaded RefreshDescriptorsAsync for single activity provider

Introduced a new overloaded method in IActivityRegistry and ActivityRegistry to refresh activity descriptors using a single IActivityProvider. Updated RefreshActivityRegistry to use the new method, ensuring more flexibility in descriptor refresh operations.
2024-09-01 09:03:14 +02:00
Sipke Schoorstra 2bbdecb22a Add Kubernetes configuration files to project sections
Included Kubernetes YAML files for elsa-server, postgres, plant-uml, and trace-lens projects in the solution. This update ensures that essential configuration files for deployments, services, and roles are managed within the project structure.
2024-08-15 11:58:39 +02:00
Sipke Schoorstra 4437d4a200 Fix build script paths and update solution file
Corrected paths in build.sh and build.ps1 to reference correct directories. Added "nuke" project to Elsa.sln for organizing build scripts under solution items.
2024-08-15 11:53:04 +02:00
Sipke Schoorstra 5bc48e6969 Remove 'nuke' project and update NUKE tool directories
Removed the 'nuke' project entry from the solution file and adjusted related NUKE tool directory paths within the build project file. These changes streamline the configuration and ensure the NUKE scripts are correctly located in the new directory structure.
2024-08-15 11:45:26 +02:00
Sipke Schoorstra 426b9bcde3 Move build and deployment scripts
This commit moves various build scripts, Docker-related files, Kubernetes deployment configurations, and migration scripts.
2024-08-14 21:42:46 +02:00
Sipke Schoorstra d308ea6a63 Fix project configuration indentation
Adjusted the indentation for the Release configuration of project {99F2B1DA-2F69-4D70-A2A3-AC985AD91EC4} in the solution file Elsa.sln. This change ensures proper alignment and readability of the project's configuration settings.
2024-08-14 21:29:05 +02:00
Sipke Schoorstra d0316687c1 Merge remote-tracking branch 'origin/patch/3.2.x' 2024-08-14 19:44:39 +02:00
Sipke Schoorstra 71d56e1665 Add ExpandoObjectConverter for JavaScript object conversion
Introduced `ConverterHelper` and `ExpandoObjectConverter` to handle conversions from .NET dynamic objects to JavaScript objects. Updated integration tests to validate the conversion and sorting functionalities.

Fixes #5895
2024-08-13 19:58:18 +02:00
Sipke Schoorstra 38998aa051
Add Orchard Core Integration + Agents Module (#5871)
* Add "integrations" project and update dependencies

A new project named "integrations" has been added to the Elsa solution. Related dependencies have been updated accordingly, with some connections reassigned to the new project. This includes both new connections established and existing connections removed in response to the project addition.

* Stash

* Refactor ActivityDescriber to clean up unused code

Removed commented out code lines related to input options in the `ActivityDescriber` class. This change enhances code readability and maintains the focus on active and necessary functionality. The main logic for getting UI properties remains intact.

* Remove deprecated webhook dispatchers and handlers

Eliminated classes and interfaces related to webhook dispatching and registration from the Elsa.Webhooks module as part of refactoring. Introduced new classes and features for handling webhook events within the Elsa.OrchardCore module. Updated several configuration files and references accordingly.

* Add OrchardCore integration sample and enhance webhooks

Added a new ASP.NET sample for integrating with OrchardCore and set up various related configuration files and project references. Improved the webhooks module by modifying the WebhookEventActivityProvider for better event handling and description management, and enhanced the WebhooksFeature to include the WebhookEventActivityProvider.

* Set activity descriptor description from eventType.DisplayName

The activity descriptor now sets its Description property using the eventType's DisplayName. This ensures that the description is consistent with the event type's display name across the system.

* Rename 'Description' to 'DisplayName' for activity descriptor

This change ensures that 'DisplayName' is assigned the correct value from 'eventType.DisplayName'. It helps avoid any confusion between 'Description' and 'DisplayName' properties in the activity descriptor object.

* Add PayloadType and Constructor logic for Webhooks

Added PayloadType to content item published events in `appsettings.json`. Updated `WebhookEventActivityProvider` to include constructor logic and manage payload types. Enhanced `OrchardCoreFeature` and `WebhooksFeature` with alias definitions for variable types.

* Add helper for webhook activity type names and refactor code to use it

Introduce `WebhookActivityTypeNameHelper` for consistent generation of webhook activity type names. Refactor relevant classes to utilize this helper, simplifying and standardizing the codebase. Adjust other minor settings, such as editor config line length and dependency configurations.

* Refactor and introduce Orchard core integration

Replace `ContentItemPublished` with a new handlers and payload structure. This change includes the creation of various classes and handlers necessary to support the OrchardCore integration, define the content item events, and configure them via app settings. This refactor aids in standardizing the handling of content item publication events through newly introduced classes and configurations.

* Refactor stimulus class name for content publication

Renamed `WebhookEventReceivedStimulus` to `ContentItemPublishedStimulus` to better reflect its purpose in handling content item publication events. Updated all references and imports accordingly to maintain consistency in the codebase.

* Refactor webhook payload handling

Switched from using only `ContentItemPublishedPayload` to incorporating both `WebhookEvent` and `ContentItemPublishedPayload` in the input dictionary. This change ensures more comprehensive data handling for webhook events.

* Add new dependency and correct project reference path

Included Elsa.Webhooks.ActivityProviders in WebhooksFeature to support new webhook activities. Corrected the project reference path in Elsa.Samples.AspNet.OrchardCoreIntegration.csproj to reflect the accurate module directory structure.

* Rename and enhance webhook handling functionality

Renamed `InvokeOrchardActivities` to `InvokeOrchardWebhookEventActivities` for clarity. Also added an `ActivityProvider` to improve webhook event handling.

* Refactor: replace Workflow with WorkflowGraph in context

Updated references from Workflow to WorkflowGraph across various files to reflect the new context structure. This change improves the accuracy of context information and aligns with recent architectural updates.

* Add Semantic Kernel modules and features

Introduced core classes and services to support Semantic Kernel agents, their configuration, and execution within workflows. Added API endpoints for agent execution, integrated kernel configuration settings, and ensured necessary extensions for skill and plugin integration.

* Update ActivityDescriber and AgentActivityProvider

Default `inputAttribute` to null in `GetUIHint` and use it in `AgentActivityProvider`. This change ensures the `GetUIHint` method is called directly, improving code reusability and simplifying input handling.

* Add GraphQL client with token authentication

Introduced a DefaultGraphQlClient to send GraphQL queries, incorporated token-based authentication using AuthenticatingDelegatingHandler, and included support for refresh tokens through DefaultSecurityTokenService. Updated projects and configuration accordingly.

* Rename OrchardWebhooks to OrchardCore

Updated method and related variables to use 'OrchardCore' instead of 'OrchardWebhooks' for consistency and improved clarity. Also added activity attributes for SendGraphQLQuery to better describe its purpose.

* Refactor webhook handling and add ProofreaderResult model

Refactor activities and event handling to support multiple webhook event types beyond content publishing. Introduce new models under OrchardCoreIntegration for ProofreaderResult and ProofreadError, and update existing methods for more generic payload handling.

* Add support for API key configuration and reference

Implemented a new `ApiKeyConfig` class to manage API keys. Updated various components to utilize API key references, enhancing configuration flexibility. Added new options and settings to facilitate these changes.

* Add translation functionality and improve agent activity displays

This commit introduces a new `TranslationResult` model and updates the project configuration to support translation features. Additionally, it adds two new project references and modifies the `appsettings.json` to include translation skills. The `AgentActivityProvider` class now has improved display names for agent activities for better readability.

* Add FactCheck and Tagging functionality

Introduced new models: FactCheckResult, GenerateTagsResult, and IncorrectFact. Integrated JetBrains.Annotations package and extended variable types in workflow management. Updated prompt templates and appsettings for proofreader, fact-checker, and tagger functions.

* Enhance 'Tagger' skill in appsettings.json

Updated the 'GenerateTags' function in the 'Tagger' skill to define the number of tags to generate and to ensure JSON response format. Added a 'Number' input variable and the 'ResponseFormat' property for clarity and precision.

* Add support for patching content items via OrchardCore API

Implement DefaultRestApiClient and IRestApiClient for GET and PATCH requests to the content API. Introduce Activity for patching content items and update app settings to handle TitleResult in JSON format.

* Add variable type aliasing and improve serialization

Introduced extension methods to add variable type and alias in WorkflowManagementFeature. Enhanced JSON serialization in AgentActivity and DefaultRestApiClient. Cleaned up solution by removing old project references and updated appsettings.json for revised prompt templates.

* Remove Proofreader models and update AgentActivity properties

Removed obsolete `ProofreaderResult` and `ProofreadError` classes to streamline the codebase. Updated `AgentActivity` properties to be internal and JSON-ignored to enhance encapsulation and reduce serialization overhead.

* Add localization support for content items in OrchardCore

Implemented a new activity `LocalizeContentItem` to create localized versions of content items based on a provided culture code. Added a corresponding API client method and request model to facilitate this feature, along with necessary format adjustments in the appsettings file.

* Enhance SkillExecutor functionality and update config details

Enabled 'AllowDangerouslySetContent' in SkillExecutor for improved flexibility with input variables. Streamlined the Proofread function prompt template and adjusted various execution settings in `appsettings.json`. Removed explicit MaxTokens limits to allow more dynamic content handling.

* Add attribute to BackgroundActivityStimulus

Added the `ExcludeFromHash` attribute to the `JobId` property in `BackgroundActivityStimulus` class, fixing background activity resumption.

* Rename SemanticKernel to Agents module for better clarity

This commit renames the SemanticKernel module to Agents across the entire project, including all relevant file paths, namespaces, and configuration settings. Major refactors encompass changing file names, class names, and namespaces to reflect the new module name, removing the SemanticKernel integration in favor of the more general Agents framework.

* Update configuration section for SemanticKernel

Changed the configuration section name from "Agents" to "SemanticKernel" in Program.cs and appsettings.json. This ensures the options are correctly bound to the relevant settings in appsettings.json.

* Add new entities and projects for Elsa Agents

Introduced new entities: `AgentDefinition`, `ApiKeyDefinition`, `ModelDefinition`, `PluginDefinition`, `SkillDefinition`. Added new projects: `Elsa.Agents.Management` and `Elsa.Agents.Persistence`. Renamed `ConfigureAgents.cs` to `ConfigureKernel.cs` and updated relevant references.

* Add ActivityHandle to workflow execution endpoint

Extended the endpoint to accept ActivityHandle as part of the workflow execution request. This change improves the ability to specify and manage activity-specific workflows. Additionally, updated `appsettings.json` to remove "Article" from the list of content types.

* Add nullable checks and new skills with functions

Improve null safety in AgentActivity and AgentActivityProvider classes by adding nullable checks. Additionally, update `appsettings.json` with corrected skill names and introduce new skills including ArticleWriter and Author.

* Register Random type in JavaScript engine

Added the Random type registration to both JintJavaScriptEvaluator and CommonTypesDefinitionProvider. This allows JavaScript code to utilize the Random class for generating random numbers.

* Add CreateContentItem activity and API client method

Introduce a new CreateContentItem activity to create and optionally publish content items. Added a new REST API client method to handle content creation, including the necessary request model.

* Update API endpoints and variable name for consistency

Changed API endpoints in DefaultRestApiClient.cs to use 'content-items' instead of 'content' for consistency with the project's URL structure. Also updated a variable name in CreateContentItem.cs for clarity and readability.

* Fix potential NullReferenceException on timer disposal.

Ensure that the timer is not null before calling Dispose in both ScheduledRecurringTask and ScheduledCronTask. This prevents potential NullReferenceExceptions when the timer event is fired.

* Add Image Generator and integrate OpenAI text-to-image feature

Introduced `ImageGenerator` for generating images from text using OpenAI's text-to-image functionality. Updated `KernelFactory` to support `OpenAITextToImage` service, and modified configurations and services to utilize the new image generation capabilities.

* Add UploadMedia activity and WorkflowInstanceStorageDriver

This commit introduces a new UploadMedia activity for uploading files to a media library and a WorkflowInstanceStorageDriver for managing workflow state storage. Additionally, it includes changes in IRestApiClient interface, DefaultRestApiClient implementation, and minor logging enhancements in ScheduledSpecificInstantTask. Unnecessary project references were removed and project files were updated accordingly.

* Mark `WorkflowStorageDriver` as obsolete and update condition.

Added [Obsolete] attribute to `WorkflowStorageDriver` indicating it will be removed in future releases. Modified `BackgroundActivityInvoker` to also recognize `WorkflowInstanceStorageDriver` for output capture.

* Handle JsonNode deserialization in ObjectConverter

Add support for deserializing JsonNode types in ObjectConverter.cs. This ensures proper handling of JsonNode along with JsonElement and JsonObject, enhancing the overall deserialization process.

* Add ResolveTags activity to OrchardCore module

Created a new ResolveTags activity that interacts with the Orchard Core API to resolve or create tags. Implemented the ResolveTagsRequest model and updated the IRestApiClient and DefaultRestApiClient to support this new functionality.

* Refactor project structure and update configuration

Renamed the Elsa.Agents project to Elsa.Agents.Activities and updated all relevant namespaces and file paths. Added a new setting to the appsettings.json to include the language parameter for article generation.

* Update project metadata in multiple csproj files

Revised descriptions and package tags across several projects to ensure consistent terminology related to Agents. Removed unused folder references in Elsa.Agents.Core.csproj.

* Simplify JSON node deserialization logic

Removed redundant JsonObject checks and unified the handling of JsonNode instances. This refactoring streamlines the code and ensures consistency in the deserialization process.
2024-08-08 19:35:43 +02:00
Sipke Schoorstra 604e5e294a Refactor role and RBAC configurations
Moved role and RBAC files from trace-lens to elsa-server directory. Updated solution file to reflect these changes and adjusted package version for WebhooksCore. This refactor improves organization and clarity in deployment configurations.
2024-07-26 11:34:30 +02:00