elsa-core/src/bundles/Elsa.Server.Web/Program.cs

430 lines
17 KiB
C#
Raw Normal View History

using System.Text.Encodings.Web;
using Elsa.Alterations.Extensions;
using Elsa.Alterations.MassTransit.Extensions;
Fix Race Condition in WorkflowDefinitionActivity (#5315) * Refactor hash method in Hasher.cs The Hash method in the Hasher.cs file has been refactored to use more efficient code. Instead of creating a SHA256 instance, it directly hashes the data using SHA256.HashData. As a result, the private Hash method and its usage, which only has relevance to the previous approach, have been removed. * Replace GetHashCode with SHA256 for script caching Two unused namespaces, Esprima and Esprima.Ast, have been removed, and two new ones, System.Security.Cryptography and System.Text, have been added. These changes in Elsa.JavaScript's JintJavaScriptEvaluator file were made to replace the basic GetHashCode method, which was used earlier to generate a cache key for JavaScript scripts, with SHA256 hash function to ensure uniqueness and avoid possible hash collisions. This greatly increases the reliability of the caching mechanism. * Set HTTP client timeout in WorkflowServer The client timeout in WorkflowServer has been set to 1 minute for `ResrService` and `client` methods. This was done to manage long running requests and prevent timeouts. * Refactor code and add validation in ScheduleActivityAsync methods An unnecessary line of code in the ScheduleActivityAsync method under the ActivityExecutionContext file has been removed to simplify the function. Meanwhile, validation has been added to ensure that specified activities are part of the workflow. This improves code clarity and prevents potential errors caused by incorrect activity scheduling. * Refactor ActivityVisitor to use ActivityVisitorContext The ActivityVisitor class in Elsa.Workflows.Core was refactored to use an ActivityVisitorContext class. The introduction of this context class replaced the multiple hashsets that were used within function signatures, consolidating them into a single object and reducing function complexity. This change improves readability and code organization. * Refactor WorkflowDefinitionActivity to use WorkflowGraph Adjusted the WorkflowDefinitionActivity class in the Elsa.Workflows.Management module. The changes include migration from using Workflow to WorkflowGraph objects and condensing code blocks for clearer readability. Additionally, a new 'IsInitialized' field is added to eliminate potential race conditions during the graph construction process. * Add activity validation to WorkflowExecutionContext A check has been implemented in the WorkflowExecutionContextExtensions to validate that a specified activity is part of the workflow. This prevents incorrect activity references when scheduling tasks in the workflow. * Refactor to use WorkflowGraph instead of Workflow The codebase has been refactored to utilize the WorkflowGraph instead of the Workflow while running and manipulating workflows. Additional changes include restructuring WorkflowRunner and WorkflowDefinitionService classes, introducing WorkflowGraphBuilder usage, and mapping updates in WorkflowDefinitionMapper. The WorkflowHost, WorkflowDispatcher, and WorkflowExecutionContext have also been updated accordingly. * Add WorkflowGraph and related services This commit introduces the IWorkflowGraphBuilder interface, the WorkflowGraph model, and an implementation of the interface in the WorkflowGraphBuilder class. The purpose of these additions is to establish the building and structure of a workflow graph. The WorkflowGraph model also includes activity node handling and hashing capabilities. * Add tests to ensure exception when scheduling an activity not part of workflow Several unit tests were written to ensure that the right behavior is exhibited when scheduling an activity that is not part of the workflow. An exception is expected to be thrown in this case. Additionally, new service definitions, workflow definitions and workflow queries were added for more comprehensive testing. * Update WorkflowGraph class and add comment descriptions This commit updates the WorkflowGraph class by extending its descriptions and implementing an explicit mention to the Workflow reference. Additionally, more attribute descriptions have been added to increase code readability and comprehension. * Removed unnecessary brackets * Remove unused services from workflow management This commit deletes "ExpressionDescriptorRegistryPopulator.cs" and "ScopedWorkflowDefinitionLookup.cs" files from Elsa.Workflows.Management.Services. These files, containing obsolete services, are no longer used in the workflow management process. The services' registration has been removed as well from "WorkflowManagementFeature.cs". * Remove TestWorkflowDefinitionService.cs from component tests A file, TestWorkflowDefinitionService.cs, was removed under Elsa.Workflows.ComponentTests. This is part of the improvement process where inefficient or unnecessary test files are cleaned up. * Add comments to WorkflowDefinitionActivityTests This commit adds explanatory comments to the WorkflowDefinitionActivityTests file. The comments provide information about the purpose of these tests and a reference to a related issue in the project's Github repository. * Updated duplicate and missing package references --------- Co-authored-by: Raymond den Haan <raymond.den.haan@nexxbiz.io>
2024-05-01 12:53:37 +00:00
using Elsa.Caching.Options;
using Elsa.Common.DistributedLocks.Noop;
using Elsa.Dapper.Extensions;
using Elsa.Dapper.Services;
using Elsa.DropIns.Extensions;
using Elsa.EntityFrameworkCore.Extensions;
using Elsa.EntityFrameworkCore.Modules.Alterations;
2023-06-28 20:46:58 +00:00
using Elsa.EntityFrameworkCore.Modules.Identity;
using Elsa.EntityFrameworkCore.Modules.Management;
using Elsa.EntityFrameworkCore.Modules.Runtime;
using Elsa.Extensions;
Add Component Testing Framework (#5261) * Add application component tests Multiple new test files were added to deliver application component tests. This move improves testing by adding integration tests that cover overall system behavior and checking end-to-end actions. Ensuring the system functions correctly as a whole. In the process, updating some package versions to maintain compatibility. * Add RefitSettings helper and revise API client service configuration The commit introduces a 'RefitSettingsHelper' for Elsa API client and revises the way the API client services are configured. It also makes improvements to the WorkflowServerTestWebAppFactory for component testing. Some endpoint contracts related to workflow execution are also updated to have optional parameters. * Remove old tests and add new workflow tests This commit removes old, unnecessary tests and incorporates new workflow tests. It also improves the Elsa API client JSON serializer and adds a helper for HttpResponseMessage. Lastly, the commit introduces changes to properly configure the test logging and to manage application settings. * Add HttpHelloWorld workflow tests A new component test scenario, HttpHelloWorldTests, has been created for testing an HttpHelloWorld workflow. This involves asserting if a workflow responds correctly with "Hello World". Furthermore, an HTTP workflow client has been introduced in the WorkflowServerTestWebAppFactory class to provide a base address for workflow API calls. * Add new test file and update workflow execution tests This change adds a new test file "fork-1.json" to the Elsa.Workflows.Api.ComponentTests project. Also, updates were made throughout the tests to replace the WorkflowServerTestWebAppFactory with a fixture, allowing the tests to run in parallel. Lastly, unnecessary warning suppression was removed from the Elsa.Workflows.Core extension method. * Add filter for .json and .elsa files in BlobStorageWorkflowProvider This change adds a BrowseFilter in the BlobStorageWorkflowProvider options. This filter checks for files that end with .json or .elsa and includes only these files when browsing through the blob storage. This filter helps prioritize specific workflow file types. * Rename WorkflowServerTestWebAppFactoryFixture and update usage The old class name "WorkflowServerTestWebAppFactoryFixture" has been replaced with the more accurate "WorkflowServerWebAppFactoryFixture". All references to the previous name in other classes were also updated accordingly. In addition, the directory key in the method "CreateConvoyOptionsBuilder" has been updated from "Workflows" to "Scenarios". * Update test fixture in workflow tests The commit updates the test fixture in two test classes: HttpHelloWorldTests and HelloWorldTests. The former test fixture, WorkflowServerTestWebAppFactoryFixture, was replaced by WorkflowServerWebAppFactoryFixture to accurately match the testing needs. * Update .csproj file paths and reorganize tests The commit modifies the file paths for several test scenario files in the Elsa.Workflows.Api.ComponentTests.csproj, reflecting a reorganization of the tests. Previously static paths have been updated to new paths under 'Scenarios'. Additionally, two new test files related to 'LogPersistenceModes' have been included in the project. * Add tests for log persistence modes This commit introduces two new test scenarios for logging persistence modes and includes a related test called 'HelloWorldWorkflow'. These tests cover scenarios where certain workflow inputs should be stored and others shouldn't, thereby testing the log persistence feature. This ensures that the logging behavior respects the specified persistence mode. * Add log persistence tests and update LogPersistenceMode enum The commit contains the addition of new log persistence tests for verifying correctness of log persistence behavior. Furthermore, the LogPersistenceMode enum has been updated, replacing 'Default' with 'Inherit'. This change makes the mode's purpose clearer. Lastly, new test scenarios and test data files were added for more comprehensive testing. * Remove obsolete component tests and support files The files removed are no longer necessary for the current state of the application. They include various component tests and their related support files within the Elsa.Workflows.Api.ComponentTests project. By removing these, the project structure is cleaner and only contains relevant tests. * Add dispatch workflow scenario tests and necessary helper classes This commit includes two new tests for dispatching workflows, along with the creation of new 'ChildWorkflow' and 'DispatchAndWaitWorkflow' classes. Auxiliary helpers and services have been added to aid in managing workflow events and signals for these tests. The 'ComponentTest' has also been upgraded to support disposal handling. * Remove ITestOutputHelper dependency from test classes Removed the dependency on ITestOutputHelper in multiple test classes across various workflow scenarios. This change simplifies the test class constructors by reducing the number of required dependencies, contributing to cleaner and leaner code. * Add 'Hello World' scenario to WorkflowCompletion tests The 'Hello World' scenario was moved into WorkflowCompletion tests, along with changes in workflow definition identifiers. As part of these changes, the 'hello-world.json' file was updated; a new file under the same name was created in the WorkflowCompletion area and the workflow identifiers in basic and workflow completion tests were updated accordingly. Additionally, 'fork-1.json' has been renamed to 'fork.json'. * Add support for cluster hosting tests This commit introduces a suite of integration tests designed to validate the behaviour of hosting multiple instances of Elsa in a clustered environment. These tests simulate a typical clustered hosting scenario by using 'App', 'Cluster', and 'Infrastructure' objects to emulate different instances of the Elsa workflow engine running on separate servers. Name changes were made to certain classes and methods to reflect their new scopes and roles within the testing environment. * Add performance tests and improve component tests Added a new performance tests project scaffold, complete with its own project file, build properties file, and a dummy test. Updated component tests to improve multi-pod testing, primarily through the addition of additional service scopes and asserting activity registry synchronization. These changes also required updates to existing project and props files as well as the solution file. * Update ActivityRegistrySyncTests and Infrastructure Added a reference to Services in ActivityRegistrySyncTests and removed unnecessary whitespace in both files. The test component Elsa.Workflows has been modified to import newly added services, ensuring all tests are running with the expected resources and services. * Fix comment * Add NOOP implementations for stores * Update PostgreSQL image and adjust test timings The PostgreSQL image used for testing has been updated to the latest version from 13.3-alpine. Timeouts in ISignalManager and DispatchWorkflowsTests have been reduced for efficiency. A delay in the ChildWorkflow has also been decreased. Additionally, an 'ImportWorkflowActivity' test in ActivityRegistrySyncTests has been marked as not yet implemented.
2024-04-26 13:49:14 +00:00
using Elsa.Features.Services;
using Elsa.Http.Options;
Implemented distributed cache refresh for workflow definitions (#5095) * Moved handler to correct namespace * Added version deletion messages to activity registry handler * Add MassTransit support for workflow management * Moved logic from Workflow.Management.MassTransit project into Elsa.MassTransit project * Added notifier to MT project for creating distributed messages * Rename and update workflow consumer class Renamed `WorkflowDefinitionConsumer` to `WorkflowDefinitionEventsConsumer` to better reflect its purpose. Updated references accordingly in the `MassTransitWorkflowManagementFeature` class to align with the new name. * Shortened instance (queue) names * Remove unnecessary comment in handler definition An unnecessary comment mark ("/") was removed in the definition of the DistributedWorkflowDefinitionNotificationsHandler class. This change contributes to code cleanup and increases code readability. * Add additional destinations in LoadBalancer settings Three new destinations have been added to the LoadBalancer settings in Elsa Server. These are primarily intended to provide broader network coverage and improve overall server performance. * Add MassTransit support in Elsa.Server A new using directive for Elsa.MassTransit.Extensions has been included at the start of the script. In addition, under certain conditions, the system now uses the MassTransit Dispatcher in the workflow management system. These changes provide support for using MassTransit in the Elsa.Server component. * Add MassTransitBroker enum for MassTransit setup A new enum called MassTransitBroker has been added to represent different types of messaging brokers used in MassTransit. This has also been integrated into the Web project's Program.cs file to allow conditional usage of brokers in the MassTransit setup, enabling more flexible and dynamic configuration. * Refactor WorkflowManagementFeature class The WorkflowManagementFeature class has been cleaned up and its formatting has been corrected. A blank line was removed, a line break was added for readability in the AddVariableType method, and several stray spaces were also removed. * Refactor RefreshActivityRegistryHandler.cs This commit removes unnecessary whitespace and improves the code readability in RefreshActivityRegistryHandler.cs. The changes include deletion of extra lines and adjustment of indentation. --------- Co-authored-by: Sipke Schoorstra <sipkeschoorstra@outlook.com>
2024-03-24 20:09:51 +00:00
using Elsa.MassTransit.Extensions;
using Elsa.MongoDb.Extensions;
using Elsa.MongoDb.Modules.Identity;
using Elsa.MongoDb.Modules.Management;
using Elsa.MongoDb.Modules.Runtime;
Implement dispatch channels (#4949) * Add DispatchWorkflowOptions and update MassTransit configuration Introduced a new class `DispatchWorkflowOptions` to provide workflow dispatch options. Updated MassTransit configuration to include NET6.0 and NET7.0 support, and ensure correct MassTransit version usage per target framework version. * Add support for configurable MassTransit message dispatching Implemented a feature which allows for configurable MassTransit message dispatching. Added support for specifying channels and message brokers. Message dispatching code was massively refactored and relevant endpoints and response models were updated to support new message dispatching features. * Add IEndpointChannelFormatter interface and implementation An interface for formatting channel queue names, 'IEndpointChannelFormatter', has been added, along with its default implementation 'DefaultEndpointChannelFormatter'. The code that uses hardcoded queue name formatting has been modified to use the new formatter instead, making it more configurable and reusable. The implementation of the formatter uses the 'Humanizer' library to kebab-case the channel names. * Add channel dispatch option to workflow activities Introduced `WorkflowDispatcherChannelOptionsProvider` to provide dropdown channel options for workflow dispatch-related activities. Updated `DispatchWorkflow` and `BulkDispatchWorkflows` activities to include a new dropdown input for specifying a dispatch channel. Also, included the selected channel name in the `DispatchWorkflowOptions` during workflow dispatching process. * Update MassTransit configurations and remove unused code The MassTransit setup in Elsa.MassTransit module has been simplified by removing conditional code for different .NET versions. Additionally, unused parameter '__X_Channel' in 'DispatchWorkflowDefinition' was removed. Lastly, the MassTransit broker in Elsa.Server.Web was switched from RabbitMq to AzureServiceBus. * Refactor dispatch workflow classes and methods Simplified the class, method and variable names related to workflow dispatching in the Elsa.Workflows.Runtime module. For example, the 'WorkflowDispatcherChannelDescriptor' class was renamed to 'DispatcherChannel'. This refactoring was performed to make code more readable and maintainable by removing redundant wording in the naming convention. * Update GitHub Workflow to support feature and issue branches The workflow changes add support for feature and issue branches. Now, it extracts the branch name and verifies the commit exists in the given branch rather than just 'main'. The versioning scheme is also modified to include the branch name and not just the run number. * Add 'bug/*' to triggering branches in packages workflow The 'bug/*' pattern was missing from the triggers that initiate the GitHub actions within our packages workflow. This update includes any branch with a 'bug/' prefix to the list, allowing bug-related branches to start jobs in our CI/CD pipeline. * Remove 'issue/*' and 'bug/*' branches from packages workflow The 'issue/*' and 'bug/*' branches have been removed from the GitHub action workflow for packages. This change was made to simplify the workflow and optimize the triggering of package building. * Update GitHub workflow to handle main branch versioning This commit modifies the GitHub workflow script to accommodate changes when the branch name is "main." If the branch name is "main", a preview version is used. It also updates script execution to print the branch name for easier debugging and verifies commit existence on the correct branch instead of dispatch channels. * Enclose branch names in quotes in packages.yml The update modifies the branch names in the packages.yml GitHub Actions workflow file. The change consists of enclosing the branch names 'main' and 'feature/*' in single quotes, ensuring compatibility and preventing potential string interpretation issues. * Update GitHub workflows package configuration The workflows package configuration has been updated to specifically watch for changes on 'feature/dispatch-channels' rather than on all feature branches. This change will prevent unnecessary builds on less relevant feature branches. * Update trigger branches in packages workflow The triggering branches in the packages workflow have been updated. Previously, only changes in the 'main' and 'feature/dispatch-channels' would trigger the workflow, now any 'feature/*' branch will. This will cause more frequent and comprehensive testing. * Add 'patch/*' to workflow trigger branches This update adds 'patch/*' to the list of branches in .github/workflows/packages.yml that can trigger the workflow. It will allow the workflow to be initiated not just for main and feature branches, but also for patches. * Add 'preview/*' to workflow triggers This commit adds a new trigger for the GitHub Actions workflow. It now also responds to push events on 'preview/*' branches, allowing for automated testing and building of these preview branches. * Update branch name extraction in GitHub Actions The extraction of the branch name has been slightly modified in the packages.yml GitHub workflow file. This alteration ensures the correct branch name is obtained for further processing within the workflow without any discrepancy. * Update branch name extraction in packages.yml Corrected the syntax for extracting the branch name within the packages.yml github workflow file. Added an extra line to print out the ref which might be useful for debugging. * Update branch name extraction in GitHub workflow The commit simplifies the way the branch name is being extracted from the GitHub ref in the packages.yml workflow file. The new method employs straightforward string manipulation, making it easier to understand and debug in case of potential issues. * Add extraction of branch name in workflow Added a new line in the GitHub workflow file (.github/workflows/packages.yml) to extract the last part after the final slash from the branch name. This enhancement allows cleaner naming conventions, especially in cases where branches are named feature/issue-123, as it will only retain 'issue-123'. * Update package naming in Github workflow The Github workflow has been updated to handle package naming more effectively. Previously, the branch name was used directly for package versioning. Now, the last part of the branch name is extracted and used as the package prefix. If the branch name is "main", the package prefix is set to "preview". * Move and add environment variable assignments The placement of the assignment for BRANCH_NAME environment variable was moved for better readability. Additionally, the PACKAGE_PREFIX environment variable was also added. These environment variables are crucial for subsequent steps in the GitHub workflow. * Add workflow dispatch validation and response handling Removed several specific dispatch response classes and consolidated all types of dispatch responses into a single DispatchWorkflowResponse class. Added a new ValidatingWorkflowDispatcher service to validate dispatch requests before they're sent. Updated several classes to work with these changes, including the BackgroundWorkflowDispatcher, MassTransitWorkflowDispatcher, and the API endpoint class. * Handle dispatch workflow failures with exceptions The DispatchWorkflow and BulkDispatchWorkflows activities now throw a FaultException when the dispatch operations fail. Previously, these operations were not checking for success and could fail silently. Now, an unsuccessful dispatch response results in a FaultException with an error message from the dispatch response.
2024-02-16 09:56:30 +00:00
using Elsa.Server.Web;
Add Component Testing Framework (#5261) * Add application component tests Multiple new test files were added to deliver application component tests. This move improves testing by adding integration tests that cover overall system behavior and checking end-to-end actions. Ensuring the system functions correctly as a whole. In the process, updating some package versions to maintain compatibility. * Add RefitSettings helper and revise API client service configuration The commit introduces a 'RefitSettingsHelper' for Elsa API client and revises the way the API client services are configured. It also makes improvements to the WorkflowServerTestWebAppFactory for component testing. Some endpoint contracts related to workflow execution are also updated to have optional parameters. * Remove old tests and add new workflow tests This commit removes old, unnecessary tests and incorporates new workflow tests. It also improves the Elsa API client JSON serializer and adds a helper for HttpResponseMessage. Lastly, the commit introduces changes to properly configure the test logging and to manage application settings. * Add HttpHelloWorld workflow tests A new component test scenario, HttpHelloWorldTests, has been created for testing an HttpHelloWorld workflow. This involves asserting if a workflow responds correctly with "Hello World". Furthermore, an HTTP workflow client has been introduced in the WorkflowServerTestWebAppFactory class to provide a base address for workflow API calls. * Add new test file and update workflow execution tests This change adds a new test file "fork-1.json" to the Elsa.Workflows.Api.ComponentTests project. Also, updates were made throughout the tests to replace the WorkflowServerTestWebAppFactory with a fixture, allowing the tests to run in parallel. Lastly, unnecessary warning suppression was removed from the Elsa.Workflows.Core extension method. * Add filter for .json and .elsa files in BlobStorageWorkflowProvider This change adds a BrowseFilter in the BlobStorageWorkflowProvider options. This filter checks for files that end with .json or .elsa and includes only these files when browsing through the blob storage. This filter helps prioritize specific workflow file types. * Rename WorkflowServerTestWebAppFactoryFixture and update usage The old class name "WorkflowServerTestWebAppFactoryFixture" has been replaced with the more accurate "WorkflowServerWebAppFactoryFixture". All references to the previous name in other classes were also updated accordingly. In addition, the directory key in the method "CreateConvoyOptionsBuilder" has been updated from "Workflows" to "Scenarios". * Update test fixture in workflow tests The commit updates the test fixture in two test classes: HttpHelloWorldTests and HelloWorldTests. The former test fixture, WorkflowServerTestWebAppFactoryFixture, was replaced by WorkflowServerWebAppFactoryFixture to accurately match the testing needs. * Update .csproj file paths and reorganize tests The commit modifies the file paths for several test scenario files in the Elsa.Workflows.Api.ComponentTests.csproj, reflecting a reorganization of the tests. Previously static paths have been updated to new paths under 'Scenarios'. Additionally, two new test files related to 'LogPersistenceModes' have been included in the project. * Add tests for log persistence modes This commit introduces two new test scenarios for logging persistence modes and includes a related test called 'HelloWorldWorkflow'. These tests cover scenarios where certain workflow inputs should be stored and others shouldn't, thereby testing the log persistence feature. This ensures that the logging behavior respects the specified persistence mode. * Add log persistence tests and update LogPersistenceMode enum The commit contains the addition of new log persistence tests for verifying correctness of log persistence behavior. Furthermore, the LogPersistenceMode enum has been updated, replacing 'Default' with 'Inherit'. This change makes the mode's purpose clearer. Lastly, new test scenarios and test data files were added for more comprehensive testing. * Remove obsolete component tests and support files The files removed are no longer necessary for the current state of the application. They include various component tests and their related support files within the Elsa.Workflows.Api.ComponentTests project. By removing these, the project structure is cleaner and only contains relevant tests. * Add dispatch workflow scenario tests and necessary helper classes This commit includes two new tests for dispatching workflows, along with the creation of new 'ChildWorkflow' and 'DispatchAndWaitWorkflow' classes. Auxiliary helpers and services have been added to aid in managing workflow events and signals for these tests. The 'ComponentTest' has also been upgraded to support disposal handling. * Remove ITestOutputHelper dependency from test classes Removed the dependency on ITestOutputHelper in multiple test classes across various workflow scenarios. This change simplifies the test class constructors by reducing the number of required dependencies, contributing to cleaner and leaner code. * Add 'Hello World' scenario to WorkflowCompletion tests The 'Hello World' scenario was moved into WorkflowCompletion tests, along with changes in workflow definition identifiers. As part of these changes, the 'hello-world.json' file was updated; a new file under the same name was created in the WorkflowCompletion area and the workflow identifiers in basic and workflow completion tests were updated accordingly. Additionally, 'fork-1.json' has been renamed to 'fork.json'. * Add support for cluster hosting tests This commit introduces a suite of integration tests designed to validate the behaviour of hosting multiple instances of Elsa in a clustered environment. These tests simulate a typical clustered hosting scenario by using 'App', 'Cluster', and 'Infrastructure' objects to emulate different instances of the Elsa workflow engine running on separate servers. Name changes were made to certain classes and methods to reflect their new scopes and roles within the testing environment. * Add performance tests and improve component tests Added a new performance tests project scaffold, complete with its own project file, build properties file, and a dummy test. Updated component tests to improve multi-pod testing, primarily through the addition of additional service scopes and asserting activity registry synchronization. These changes also required updates to existing project and props files as well as the solution file. * Update ActivityRegistrySyncTests and Infrastructure Added a reference to Services in ActivityRegistrySyncTests and removed unnecessary whitespace in both files. The test component Elsa.Workflows has been modified to import newly added services, ensuring all tests are running with the expected resources and services. * Fix comment * Add NOOP implementations for stores * Update PostgreSQL image and adjust test timings The PostgreSQL image used for testing has been updated to the latest version from 13.3-alpine. Timeouts in ISignalManager and DispatchWorkflowsTests have been reduced for efficiency. A delay in the ChildWorkflow has also been decreased. Additionally, an 'ImportWorkflowActivity' test in ActivityRegistrySyncTests has been marked as not yet implemented.
2024-04-26 13:49:14 +00:00
using Elsa.Workflows;
using Elsa.Workflows.Management.Compression;
Add caching to workflow runtime and workflow management stores (#5174) * Add caching to workflow runtime and workflow management stores The update introduces caching to workflow runtime and workflow management stores to enhance performance. This is achieved by adding decorators for several stores, which cache records to reduce database fetches. Additionally, a signaler for change tokens allows for cache invalidation when changes occur. The MemoryCache feature has also been updated to include Scrutor for decoration and the caching duration can be configured through the new CachingOptions class. * Refactor WorkflowsMiddleware for HTTP Endpoint bookmarks and triggers The WorkflowsMiddleware has been extensively refactored to handle HTTP Endpoint bookmarks and triggers. This involves breaking down the InvokeAsync method by extracting parts of its functionality into separate helper methods such as FindTriggersAsync and FindBookmarksAsync. Moreover, Assist with authorization checks, workflow execution within request timeout, and handling of workflow faults has been improved to be more efficient and clearly segmented. * Update HTTP endpoint authorization to use Workflow context The authorization process in the AuthenticationBasedHttpEndpointAuthorizationHandler class has been updated to use the Workflow context instead of the WorkflowInstanceId string. The AuthorizeHttpEndpointContext model has been correspondingly changed to include a Workflow property, thereby strengthening the link between authorization and specific workflows. * Add FindAsync methods to trigger and bookmark stores The code adjustments add new FindAsync methods to the trigger and bookmark store contracts as well as all their concrete implementations (MongoDb, Memory and EFCore). These methods support fetching the first record matching a given filter. The adjustments also include minor syntax improvements and the addition of [UsedImplicitly] attributes where needed. * Refactor WorkflowsMiddleware for improved workflow handling The code was refactored to simplify the flow of handling workflows in the WorkflowsMiddleware class. Specifically, the methods to start and resume a workflow have been extracted to improve code readability. Further, handleErrorMiddlewares was also updated to better manage instances where no valid workflows or base paths are found. * Add caching functionality to WorkflowsMiddleware Added IMemoryCache usage in the WorkflowsMiddleware to cache lookup results for workflows and their associated triggers. This will reduce the number of database operations required when searching for workflows, thus improving performance. The cache is maintained for one minute before it is refreshed. * Implement dynamic cache duration for workflows The code has been updated to have a dynamic cache duration for the workflows instead of a hardcoded one minute. By using the CachingOptions service, the cache duration can now be set in the configuration making it more flexible and adaptable to different performance needs. * Add HttpWorkflowsCacheManager for caching HTTP workflows This commit includes the implementation of IHttpWorkflowsCacheManager for caching of HTTP workflows. New handlers have been added to invalidate cache on workflow updates. Additionally, WorkflowsMiddleware has been renamed to HttpWorkflowsMiddleware. * Refactor workflow trigger handling and caching The refactoring includes deletion of `IndexWorkflowTriggersHandler.cs` and creation of `IndexTriggers.cs` thus revising the workflow trigger indexing approach. Also, revamped the `ITriggerIndexer` interface which now handles deletion of triggers with specific workflow and filter. Furthermore, the caching mechanism in `HttpWorkflowsCacheManager.cs` is modified to handle eviction of workflow definitions and triggers separately boosting its efficiency. * Add summary to IndexedWorkflowTriggers A summary has been added to the 'IndexedWorkflowTriggers' class, providing a brief description. This description outlines that it represents a collection of indexed workflow triggers, promoting clearer understanding for future reference. * Refactor memory caching feature into separate module This commit separates the memory caching feature from the Elsa.Common module into a distinct Elsa.Caching module. This includes moving and renaming related files, such as the MemoryCacheFeature class and associated dependencies. The references in other modules and in the main solution file have been updated accordingly to include the new Elsa.Caching module. * Add distributed caching and update async methods Introduced a distributed caching feature with extensible change token signal publishing. Updated various cache-related methods to be asynchronous for improved performance and responsiveness. Also updated some workflow identity references for clarity. * Add distributed caching with MassTransit support This addition includes the implementation of a distributed caching system with MassTransit transport. The changes introduce necessary interfaces and services, new distributed caching feature along with the support for MassTransit as a transport option. Moreover, the instance management feature has been renamed to clustering feature for better clarity. * Refactor queue naming and scope of MassTransitChangeTokenSignalPublisher Queue name construction is adjusted in the RabbitMqServiceBusFeature and AzureServiceBusFeature modules for better organization and readability. MassTransitChangeTokenSignalPublisher is now a singleton service, ensuring the signal publisher can be shared across the application, increasing efficiency and performance. Also, introduced use of DistributedCacheFeature in MassTransitDistributedCacheFeature module for better modularization. * Add caching capabilities to workflow definition service This commit introduces caching to the workflow definition service, improving the performance for retrieving workflow definitions. 4 new classes have been created (`CachingWorkflowDefinitionService`, `EvictWorkflowDefinitionServiceCache`, `WorkflowDefinitionCacheManager`, and `IWorkflowDefinitionCacheManager`), and several existing classes have been updated to support caching. The caching also includes invalidation mechanisms, ensuring data consistency. * Refactor caching mechanism in workflow definition In the workflow definition module, the explicit caching functionality related to workflow definition versioning has been removed in favor of a more streamlined approach. Additionally, the manner in which services are registered has been altered. As a result, the caching now directly involves the overall workflow definition rather than individual versions, simplifying the caching logic and potentially improving the performance. * Update MassTransitBroker and enable RealTimeWorkflows and SignalRHubs The MassTransitBroker has been updated to Memory from RabbitMq. In addition, the RealTimeWorkflows and UseWorkflowsSignalRHubs features are now enabled in the code. This change will impact how the service communicates and processes real-time requests for workflow operations. * Reformat variable types in HttpFeature The reformatting involves a list of variable types in the HttpFeature module. Each type now appears on a new line for improved readability, making the code easier to maintain and review. * Update HTTP workflows cache invalidation handler XML comment * Remove unnecessary using directives Unnecessary using directives were deleted across several files in the Elsa.Http module. This simplifies the code and will possibly improve execution speed. Specific deletions include those for Encoding, Unicode, Extensions, Collections.Generic, Linq, Text, and Tasks namespaces. * Refactor HttpWorkflowsMiddleware constructor This commit simplifies the HttpWorkflowsMiddleware class constructor. It removes the intermediary variables `_next` and `_options` and directly uses the passed arguments in the constructor. Now, the `next` and `options` parameters are used directly throughout the middleware. * Simplify workflow retrieval in HttpWorkflowsMiddleware This refactoring replaces the use of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync with a single method, FindWorkflowAsync. This simplifies the middleware code and likely improves performance by reducing the number of database queries or service calls required to retrieve a workflow. * Refactor workflow retrieval in HttpBookmarkProcessor Simplified the workflow retrieval process in HttpBookmarkProcessor.cs. Replaced FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods with a single FindWorkflowAsync call. This reduces the complexity and improves efficiency in retrieving workflow. * Optimize FindWorkflowAsync method in HttpWorkflowsCacheManager Removed redundant lines of code to simplify workflow search functionality. This change simplified the FindWorkflowAsync method by directly calling the FindWorkflowAsync function in the workflowDefinitionService, thus increasing code readability and efficiency. * Refactor Endpoint.cs for workflow retrieval The method for obtaining a workflow in the Endpoint.cs script has been refactored and streamlined. The 'GetWorkflowDefinition' method is replaced by the 'GetWorkflowAsync' method which directly retrieves the workflow, without the intermediate step of materializing the workflow definition. This shortens the code and simplifies the process. * Refactor InputFunctionsDefinitionProvider constructor The constructor for InputFunctionsDefinitionProvider has been simplified by removing unnecessary private fields. Services are now directly used in the method instead of being stored in fields. This improves readability and reduces complexity in the class structure. * Refactor WorkflowInstance with improved state handling Simplified the methods for handling workflow and workflow state in the WorkflowInstance class. The refactoring also included some code clean-ups and variable renaming. The new implementation provides better readability and maintainability of the code by reducing unnecessary lines and improving structuring of objects and responses. * Remove unused IBookmarkManager and update workflow functions IBookmarkManager from ProtoActorWorkflowRuntime.cs file is removed due to its redundant status. Additionally, the "FindAsync" method has been updated to use a cancellation token. Also, annotations were added to the "ExportWorkflowStateAsync" and "ImportWorkflowStateAsync" methods to flag calls to functions that require unreferenced code. * Remove unused ReSharper directive Unused ReSharper directive in the file IndexTriggers.cs was identified and therefore removed. This change makes the code cleaner and easier to read. * Update activity invocation in workflow runtime Updated the DefaultBackgroundActivityInvoker service in the Elsa.Workflows.Runtime module to annotate the ExecuteAsync method with "RequiresUnreferencedCode" attribute. This change is made considering the potential code trimming issue. Additionally, simplified the process of fetching workflow by directly using FindWorkflowAsync method instead of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods. * Refactor code to simplify workflow definition loading The code for finding and materializing workflow definitions has been simplified. Instead of loading the definition and materializing it into a workflow in separate steps, a new method called FindWorkflowAsync has been introduced to perform both actions at once. This reduces redundancy and makes the code more readable. * Refactor WorkflowHostFactory to streamline workflow creation This commit simplifies the workflow creation process in WorkflowHostFactory. It removes redundant code and extraneous methods, specifically the overloaded CreateAsync method which used WorkflowDefinition. Now, it directly finds and uses the Workflow instance, thereby simplifying the code base and improving maintainability. * Refactor workflow retrieval in WorkflowInstance.cs Changed the way workflow instances are retrieved from the WorkflowDefinitionService. Instead of obtaining the workflow definition and then materializing the workflow from it, the workflow is directly retrieved using the FindWorkflowAsync function. This simplifies the code and avoids unnecessary null-checks. * Remove unnecessary whitespace in WorkflowInstance.cs An extraneous whitespace character was identified and removed in the WorkflowInstance.cs file. This change contributes towards maintaining clean and readable code in the Elsa.ProtoActor module. * Remove unnecessary comment in ProtoActorWorkflowRuntime The unnecessary comment ("Load the workflow definition.") in the method TryStartWorkflowAsync of the ProtoActorWorkflowRuntime.cs file was removed. This is part of an ongoing effort to keep the codebase clean and readable. * Update workflow management features and handlers Added explicit notification handlers for DeleteWorkflowInstances and RefreshActivityRegistry in WorkflowManagementFeature.cs. Also, renamed RefreshActivityRegistryHandler.cs to RefreshActivityRegistry.cs for better clarity. * Add multiple log record support to workflow execution log stores The major change of this commit is the addition of methods to add multiple log records in the WorkflowExecutionLogStore, across different storage modules such as EntityFramework, MongoDB, Dapper, Elasticsearch and Memory. This ensures consistency and uniform behavior across different storage types. Furthermore, some reformatting and tidying up of the code were undertaken to maintain readability and clarity. * Remove redundant workflow definition check The workflow definition existence check and related service retrieval were removed from HttpWorkflowsMiddleware.cs. It was determined that this check was unnecessary as the workflow definition's existence is guaranteed at this point in the process, reducing redundancy in the code. * Improve cancellation token usage in workflow execution This commit refines the usage of cancellation tokens during the execution of workflows in Elsa.Server and HttpWorkflowsMiddleware. Previously, a cancellation token pair was created before ExecuteWithinTimeoutAsync was called, which limited duration control solely to that method. Now, cancellation tokens are included within ExecuteWithinTimeoutAsync method. This allows the method to observe any cancellation initiated by outer scopes, enhancing control over the timeout of operations. * Add PersistStateAsync method to WorkflowHost A new PersistStateAsync method has been added to the WorkflowHost, which enables the host to directly persist its own state. The method has been integrated into the DefaultWorkflowRuntime and HttpWorkflowsMiddleware. This update eliminates the need to continuously get instances of IWorkflowInstanceManager to save state, which improves efficiency and code readability. * Refactor DefaultAlterationRunner service This update simplifies the DefaultAlterationRunner service, reducing the number of code lines and removing unnecessary references. The workflow materialization step has been merged with the find workflow step, and unused namespaces have been dropped. * Refine wording in IWorkflowHost interface documentation The documentation for the 'CanStartWorkflowAsync' method in the IWorkflowHost interface has been cleaned up. The superfluous "or not" verbiage has been removed, making it easier to understand the method's function. * Remove Redis from DistributedCachingTransport The Redis option was removed from the DistributedCachingTransport enumeration. This transport isn't currently implemented. * Remove 'useDistributedCaching' constant The 'useDistributedCaching' constant was removed from `Program.cs`, and conditional logic was updated to use `distributedCachingTransport != DistributedCachingTransport.None`. A new option 'None' was added to the `DistributedCachingTransport` enum to facilitate this change. * Update package tags in MassTransit project file The package tags in the Elsa.Caching.Distributed.MassTransit project file was updated to consolidate the tags, changing 'mass-transit' to 'masstransit'. This change better aligns with standard naming conventions and improves searchability. * Refactor distributed caching implementation This commit involves an extensive refactor of the distributed caching implementation. Distributed caching related code and resources were moved into an independent 'Elsa.Caching.Distributed' module. The interface 'IDistributedChangeTokenSignaler' was deleted and its functionality was replaced by 'IChangeTokenSignalInvoker'. * Refactor order of parameters in GetOrCreateAsync method The order of parameters in the GetOrCreateAsync method within the CachingWorkflowDefinitionStore class has been changed. This change ensures that the `key` parameter is now first, followed by the `factory` parameter. This improves code readability and aligns with standard coding practices. * Refactor cache retrieval in Workflow service Refactoring was done to streamline the way objects are retrieved from cache in the Workflow service. Duplicated code was condensed into a new `GetFromCacheAsync` method, which is now called in the existing methods, thus increasing maintainability and reducing the possibility of errors. * Update method descriptions and fix comments formatting Method descriptions in various contracts have been updated to more accurately reflect their function regarding record addition and updating in the persistence store. All double comment markers (/// ///) have also been corrected to the standard (///) across multiple classes. * Remove unused caching methods in ModuleExtensions The commit removes the unused methods, `UseMemoryCache` and `UseDistributedCache` from the `ModuleExtensions.cs` file. The removal is part of a wider cleanup and refactoring effort to streamline the codebase and improve legibility. * Remove redundant PrimaryKeyName in DapperWorkflowExecutionLogStore The "PrimaryKeyName" constant was removed in DapperWorkflowExecutionLogStore. This change simplifies the initialization of the '_store' property, reducing unnecessary redundancy and complexity. The refactored code maintains the same functionality but improves readability and maintainability. * Refactor SaveAsync methods in Elsa.Dapper Store The SaveAsync functions have been updated in the Store.cs file inside the Elsa.Dapper module. They now include cancellation token parameters and specify that they add or update records, providing clearer distinction and flexibility. * Refactor store initialization in Elsa.Dapper modules Removed the redundant usage of primary keys during the store initialization across Elsa.Dapper module. Simplified the SaveAsync methods by removing the parameter for primary key, making the code cleaner and more maintainable. This refactoring does not affect the module's functionality. * Refactor UserStore in Elsa.Dapper module The code was adjusted to improve readability within the Elsa.Dapper module's UserStore. Two lines that were previously combined have now been separated into distinct lines, making the code structure more clear. * Refactor constructor arguments in MongoDb module Simplified several classes in the MongoDb module by injecting dependencies directly through the constructor instead of assigning them to private readonly fields. This improves readability and removes unnecessary code lines. Also added JetBrains.Annotations where applicable. * Fix comment syntax in IWorkflowInstanceStore A syntax error in the comments for the method SaveManyAsync (in IWorkflowInstanceStore interface) has been corrected. This change ensures that the remarks section of the method is properly formatted and correctly displayed in documentation. * Remove ComputeBookmarkHash from IHttpWorkflowsCacheManager The ComputeBookmarkHash method was removed from IHttpWorkflowsCacheManager to declutter the interface. The functionality was moved and adapted in the HttpWorkflowsMiddleware class to maintain the original functionality. * Add logging to HttpWorkflowsMiddleware In this update, the HttpWorkflowsMiddleware class has been modified to include logging. Specifically, warning logs have been added to track workflow-related processes and to notify if mentioned bookmarks or workflow instances are not found. * Update consumer configuration in MassTransitFeature This commit modifies the consumer configuration in the MassTransitFeature. Instead of hardcoding the consumer type to DispatchCancelWorkflowsRequestConsumer, it now uses the dynamic consumer type retrieved from the context, making the feature more adaptable for different scenarios. * Change default MassTransitBroker to Memory The default value for the variable useMassTransitBroker in Elsa.Server.Web's Program.cs file has been modified. It has been changed from RabbitMq to Memory to change the message broker used by MassTransit in the application. * Remove Datadog.Trace package from Directory.Packages.props The Datadog.Trace package with version 2.49.0 has been removed from the Directory.Packages.props file. This change reflects the fact that this package is no longer required in our project.
2024-04-10 09:51:40 +00:00
using Elsa.Workflows.Management.Stores;
using Elsa.Workflows.Runtime.Stores;
Add Component Testing Framework (#5261) * Add application component tests Multiple new test files were added to deliver application component tests. This move improves testing by adding integration tests that cover overall system behavior and checking end-to-end actions. Ensuring the system functions correctly as a whole. In the process, updating some package versions to maintain compatibility. * Add RefitSettings helper and revise API client service configuration The commit introduces a 'RefitSettingsHelper' for Elsa API client and revises the way the API client services are configured. It also makes improvements to the WorkflowServerTestWebAppFactory for component testing. Some endpoint contracts related to workflow execution are also updated to have optional parameters. * Remove old tests and add new workflow tests This commit removes old, unnecessary tests and incorporates new workflow tests. It also improves the Elsa API client JSON serializer and adds a helper for HttpResponseMessage. Lastly, the commit introduces changes to properly configure the test logging and to manage application settings. * Add HttpHelloWorld workflow tests A new component test scenario, HttpHelloWorldTests, has been created for testing an HttpHelloWorld workflow. This involves asserting if a workflow responds correctly with "Hello World". Furthermore, an HTTP workflow client has been introduced in the WorkflowServerTestWebAppFactory class to provide a base address for workflow API calls. * Add new test file and update workflow execution tests This change adds a new test file "fork-1.json" to the Elsa.Workflows.Api.ComponentTests project. Also, updates were made throughout the tests to replace the WorkflowServerTestWebAppFactory with a fixture, allowing the tests to run in parallel. Lastly, unnecessary warning suppression was removed from the Elsa.Workflows.Core extension method. * Add filter for .json and .elsa files in BlobStorageWorkflowProvider This change adds a BrowseFilter in the BlobStorageWorkflowProvider options. This filter checks for files that end with .json or .elsa and includes only these files when browsing through the blob storage. This filter helps prioritize specific workflow file types. * Rename WorkflowServerTestWebAppFactoryFixture and update usage The old class name "WorkflowServerTestWebAppFactoryFixture" has been replaced with the more accurate "WorkflowServerWebAppFactoryFixture". All references to the previous name in other classes were also updated accordingly. In addition, the directory key in the method "CreateConvoyOptionsBuilder" has been updated from "Workflows" to "Scenarios". * Update test fixture in workflow tests The commit updates the test fixture in two test classes: HttpHelloWorldTests and HelloWorldTests. The former test fixture, WorkflowServerTestWebAppFactoryFixture, was replaced by WorkflowServerWebAppFactoryFixture to accurately match the testing needs. * Update .csproj file paths and reorganize tests The commit modifies the file paths for several test scenario files in the Elsa.Workflows.Api.ComponentTests.csproj, reflecting a reorganization of the tests. Previously static paths have been updated to new paths under 'Scenarios'. Additionally, two new test files related to 'LogPersistenceModes' have been included in the project. * Add tests for log persistence modes This commit introduces two new test scenarios for logging persistence modes and includes a related test called 'HelloWorldWorkflow'. These tests cover scenarios where certain workflow inputs should be stored and others shouldn't, thereby testing the log persistence feature. This ensures that the logging behavior respects the specified persistence mode. * Add log persistence tests and update LogPersistenceMode enum The commit contains the addition of new log persistence tests for verifying correctness of log persistence behavior. Furthermore, the LogPersistenceMode enum has been updated, replacing 'Default' with 'Inherit'. This change makes the mode's purpose clearer. Lastly, new test scenarios and test data files were added for more comprehensive testing. * Remove obsolete component tests and support files The files removed are no longer necessary for the current state of the application. They include various component tests and their related support files within the Elsa.Workflows.Api.ComponentTests project. By removing these, the project structure is cleaner and only contains relevant tests. * Add dispatch workflow scenario tests and necessary helper classes This commit includes two new tests for dispatching workflows, along with the creation of new 'ChildWorkflow' and 'DispatchAndWaitWorkflow' classes. Auxiliary helpers and services have been added to aid in managing workflow events and signals for these tests. The 'ComponentTest' has also been upgraded to support disposal handling. * Remove ITestOutputHelper dependency from test classes Removed the dependency on ITestOutputHelper in multiple test classes across various workflow scenarios. This change simplifies the test class constructors by reducing the number of required dependencies, contributing to cleaner and leaner code. * Add 'Hello World' scenario to WorkflowCompletion tests The 'Hello World' scenario was moved into WorkflowCompletion tests, along with changes in workflow definition identifiers. As part of these changes, the 'hello-world.json' file was updated; a new file under the same name was created in the WorkflowCompletion area and the workflow identifiers in basic and workflow completion tests were updated accordingly. Additionally, 'fork-1.json' has been renamed to 'fork.json'. * Add support for cluster hosting tests This commit introduces a suite of integration tests designed to validate the behaviour of hosting multiple instances of Elsa in a clustered environment. These tests simulate a typical clustered hosting scenario by using 'App', 'Cluster', and 'Infrastructure' objects to emulate different instances of the Elsa workflow engine running on separate servers. Name changes were made to certain classes and methods to reflect their new scopes and roles within the testing environment. * Add performance tests and improve component tests Added a new performance tests project scaffold, complete with its own project file, build properties file, and a dummy test. Updated component tests to improve multi-pod testing, primarily through the addition of additional service scopes and asserting activity registry synchronization. These changes also required updates to existing project and props files as well as the solution file. * Update ActivityRegistrySyncTests and Infrastructure Added a reference to Services in ActivityRegistrySyncTests and removed unnecessary whitespace in both files. The test component Elsa.Workflows has been modified to import newly added services, ensuring all tests are running with the expected resources and services. * Fix comment * Add NOOP implementations for stores * Update PostgreSQL image and adjust test timings The PostgreSQL image used for testing has been updated to the latest version from 13.3-alpine. Timeouts in ISignalManager and DispatchWorkflowsTests have been reduced for efficiency. A delay in the ChildWorkflow has also been decreased. Additionally, an 'ImportWorkflowActivity' test in ActivityRegistrySyncTests has been marked as not yet implemented.
2024-04-26 13:49:14 +00:00
using JetBrains.Annotations;
using Medallion.Threading.FileSystem;
using Medallion.Threading.Postgres;
using Medallion.Threading.Redis;
2023-07-23 19:05:31 +00:00
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Options;
2023-07-23 19:05:31 +00:00
using Proto.Persistence.Sqlite;
using Proto.Persistence.SqlServer;
const bool useMongoDb = false;
const bool useSqlServer = false;
const bool usePostgres = false;
const bool useCockroachDb = false;
const bool useDapper = false;
2024-01-03 08:30:24 +00:00
const bool useProtoActor = false;
const bool useHangfire = false;
const bool useQuartz = true;
const bool useMassTransit = true;
const bool useZipCompression = true;
const bool runEFCoreMigrations = true;
Refactor workflow dispatch and instance creation process (#5213) * Refactor workflow dispatch and instance creation process This update splits the process of dispatching a workflow into two steps: Initialization and Execution. Now, first, a new workflow instance is created and saved with the input parameters. Second, the new workflow instance is dispatched for execution. This process ensures that the size of the message dispatched does not exceed acceptable limits and enhances workflow dispatch efficiency. It also helps avoid data loss in case of premature process termination or failure in the initial stages of execution. * Change default workflow substatus to 'Pending' The code changes involve modifying the default WorkflowSubStatus from 'Executing' to 'Pending'. This minor adjustment is implemented to represent a more accurate initial state of a new workflow instance in the WorkflowManagement module in Elsa. * Add support for existing workflow instances in WorkflowInstance grain The change extends the WorkflowInstance grain to include an IWorkflowInstanceStore to support workflows that already exist. Additionally, the CreateWorkflowHostAsync method is enhanced to rebuild a workflow host if the instance already exists. * Simplify XML comments format * Refactor comments in `StartWorkflowHostParams` class Removed unnecessary comment tags in the `StartWorkflowHostParams` class for better readability and simplicity. Also, unused lines of code from the `IWorkflowRuntime` interface have been commented out to enhance clarity and cleanliness in the codebase. * Remove unnecessary comment in MassTransitWorkflowDispatcher The obsolete comment about attaching a version header to the message was removed from 'MassTransitWorkflowDispatcher.cs'. This comment was no longer relevant as the dispatcher no longer performs the action described.
2024-04-12 08:46:40 +00:00
const bool useMemoryStores = false;
const bool useCaching = true;
const bool useReadOnlyMode = false;
const DistributedCachingTransport distributedCachingTransport = DistributedCachingTransport.MassTransit;
const MassTransitBroker useMassTransitBroker = MassTransitBroker.RabbitMq;
2023-05-16 18:03:12 +00:00
var builder = WebApplication.CreateBuilder(args);
var services = builder.Services;
var configuration = builder.Configuration;
var identitySection = configuration.GetSection("Identity");
var identityTokenSection = identitySection.GetSection("Tokens");
var sqliteConnectionString = configuration.GetConnectionString("Sqlite")!;
var sqlServerConnectionString = configuration.GetConnectionString("SqlServer")!;
var postgresConnectionString = configuration.GetConnectionString("PostgreSql")!;
var cockroachDbConnectionString = configuration.GetConnectionString("CockroachDb")!;
var mongoDbConnectionString = configuration.GetConnectionString("MongoDb")!;
var azureServiceBusConnectionString = configuration.GetConnectionString("AzureServiceBus")!;
var rabbitMqConnectionString = configuration.GetConnectionString("RabbitMq")!;
var redisConnectionString = configuration.GetConnectionString("Redis")!;
var distributedLockProviderName = configuration.GetSection("Runtime")["DistributedLockProvider"];
2022-08-27 19:58:22 +00:00
// Add Elsa services.
services
.AddElsa(elsa =>
{
if (useMongoDb)
elsa.UseMongoDb(mongoDbConnectionString);
if (useDapper)
elsa.UseDapper(dapper =>
{
dapper.UseMigrations(feature =>
{
if (useSqlServer)
feature.UseSqlServer();
else
feature.UseSqlite();
});
dapper.DbConnectionProvider = sp =>
{
if (useSqlServer)
return new SqlServerDbConnectionProvider(sqlServerConnectionString!);
else
return new SqliteDbConnectionProvider(sqliteConnectionString);
};
});
if (useHangfire)
elsa.UseHangfire();
elsa
.AddActivitiesFrom<Program>()
.AddWorkflowsFrom<Program>()
.UseFluentStorageProvider()
.UseFileStorage()
.UseIdentity(identity =>
{
if (useMongoDb)
identity.UseMongoDb();
else if (useDapper)
identity.UseDapper();
else
identity.UseEntityFrameworkCore(ef =>
{
if (useSqlServer)
ef.UseSqlServer(sqlServerConnectionString!);
else if (usePostgres)
ef.UsePostgreSql(postgresConnectionString!);
else if (useCockroachDb)
ef.UsePostgreSql(cockroachDbConnectionString!);
else
ef.UseSqlite(sqliteConnectionString);
ef.RunMigrations = runEFCoreMigrations;
});
identity.IdentityOptions = options => identitySection.Bind(options);
identity.TokenOptions = options => identityTokenSection.Bind(options);
identity.UseConfigurationBasedUserProvider(options => identitySection.Bind(options));
identity.UseConfigurationBasedApplicationProvider(options => identitySection.Bind(options));
identity.UseConfigurationBasedRoleProvider(options => identitySection.Bind(options));
})
.UseDefaultAuthentication()
.UseWorkflowManagement(management =>
{
if (useMongoDb)
management.UseMongoDb();
else if (useDapper)
management.UseDapper();
else
management.UseEntityFrameworkCore(ef =>
{
if (useSqlServer)
ef.UseSqlServer(sqlServerConnectionString!);
else if (usePostgres)
ef.UsePostgreSql(postgresConnectionString!);
else if (useCockroachDb)
ef.UsePostgreSql(cockroachDbConnectionString!);
else
ef.UseSqlite(sqliteConnectionString);
ef.RunMigrations = runEFCoreMigrations;
});
if (useZipCompression)
management.SetCompressionAlgorithm(nameof(Zstd));
if (useMemoryStores)
management.UseWorkflowInstances(feature => feature.WorkflowInstanceStore = sp => sp.GetRequiredService<MemoryWorkflowInstanceStore>());
if (useMassTransit)
Implemented distributed cache refresh for workflow definitions (#5095) * Moved handler to correct namespace * Added version deletion messages to activity registry handler * Add MassTransit support for workflow management * Moved logic from Workflow.Management.MassTransit project into Elsa.MassTransit project * Added notifier to MT project for creating distributed messages * Rename and update workflow consumer class Renamed `WorkflowDefinitionConsumer` to `WorkflowDefinitionEventsConsumer` to better reflect its purpose. Updated references accordingly in the `MassTransitWorkflowManagementFeature` class to align with the new name. * Shortened instance (queue) names * Remove unnecessary comment in handler definition An unnecessary comment mark ("/") was removed in the definition of the DistributedWorkflowDefinitionNotificationsHandler class. This change contributes to code cleanup and increases code readability. * Add additional destinations in LoadBalancer settings Three new destinations have been added to the LoadBalancer settings in Elsa Server. These are primarily intended to provide broader network coverage and improve overall server performance. * Add MassTransit support in Elsa.Server A new using directive for Elsa.MassTransit.Extensions has been included at the start of the script. In addition, under certain conditions, the system now uses the MassTransit Dispatcher in the workflow management system. These changes provide support for using MassTransit in the Elsa.Server component. * Add MassTransitBroker enum for MassTransit setup A new enum called MassTransitBroker has been added to represent different types of messaging brokers used in MassTransit. This has also been integrated into the Web project's Program.cs file to allow conditional usage of brokers in the MassTransit setup, enabling more flexible and dynamic configuration. * Refactor WorkflowManagementFeature class The WorkflowManagementFeature class has been cleaned up and its formatting has been corrected. A blank line was removed, a line break was added for readability in the AddVariableType method, and several stray spaces were also removed. * Refactor RefreshActivityRegistryHandler.cs This commit removes unnecessary whitespace and improves the code readability in RefreshActivityRegistryHandler.cs. The changes include deletion of extra lines and adjustment of indentation. --------- Co-authored-by: Sipke Schoorstra <sipkeschoorstra@outlook.com>
2024-03-24 20:09:51 +00:00
management.UseMassTransitDispatcher();
Add caching to workflow runtime and workflow management stores (#5174) * Add caching to workflow runtime and workflow management stores The update introduces caching to workflow runtime and workflow management stores to enhance performance. This is achieved by adding decorators for several stores, which cache records to reduce database fetches. Additionally, a signaler for change tokens allows for cache invalidation when changes occur. The MemoryCache feature has also been updated to include Scrutor for decoration and the caching duration can be configured through the new CachingOptions class. * Refactor WorkflowsMiddleware for HTTP Endpoint bookmarks and triggers The WorkflowsMiddleware has been extensively refactored to handle HTTP Endpoint bookmarks and triggers. This involves breaking down the InvokeAsync method by extracting parts of its functionality into separate helper methods such as FindTriggersAsync and FindBookmarksAsync. Moreover, Assist with authorization checks, workflow execution within request timeout, and handling of workflow faults has been improved to be more efficient and clearly segmented. * Update HTTP endpoint authorization to use Workflow context The authorization process in the AuthenticationBasedHttpEndpointAuthorizationHandler class has been updated to use the Workflow context instead of the WorkflowInstanceId string. The AuthorizeHttpEndpointContext model has been correspondingly changed to include a Workflow property, thereby strengthening the link between authorization and specific workflows. * Add FindAsync methods to trigger and bookmark stores The code adjustments add new FindAsync methods to the trigger and bookmark store contracts as well as all their concrete implementations (MongoDb, Memory and EFCore). These methods support fetching the first record matching a given filter. The adjustments also include minor syntax improvements and the addition of [UsedImplicitly] attributes where needed. * Refactor WorkflowsMiddleware for improved workflow handling The code was refactored to simplify the flow of handling workflows in the WorkflowsMiddleware class. Specifically, the methods to start and resume a workflow have been extracted to improve code readability. Further, handleErrorMiddlewares was also updated to better manage instances where no valid workflows or base paths are found. * Add caching functionality to WorkflowsMiddleware Added IMemoryCache usage in the WorkflowsMiddleware to cache lookup results for workflows and their associated triggers. This will reduce the number of database operations required when searching for workflows, thus improving performance. The cache is maintained for one minute before it is refreshed. * Implement dynamic cache duration for workflows The code has been updated to have a dynamic cache duration for the workflows instead of a hardcoded one minute. By using the CachingOptions service, the cache duration can now be set in the configuration making it more flexible and adaptable to different performance needs. * Add HttpWorkflowsCacheManager for caching HTTP workflows This commit includes the implementation of IHttpWorkflowsCacheManager for caching of HTTP workflows. New handlers have been added to invalidate cache on workflow updates. Additionally, WorkflowsMiddleware has been renamed to HttpWorkflowsMiddleware. * Refactor workflow trigger handling and caching The refactoring includes deletion of `IndexWorkflowTriggersHandler.cs` and creation of `IndexTriggers.cs` thus revising the workflow trigger indexing approach. Also, revamped the `ITriggerIndexer` interface which now handles deletion of triggers with specific workflow and filter. Furthermore, the caching mechanism in `HttpWorkflowsCacheManager.cs` is modified to handle eviction of workflow definitions and triggers separately boosting its efficiency. * Add summary to IndexedWorkflowTriggers A summary has been added to the 'IndexedWorkflowTriggers' class, providing a brief description. This description outlines that it represents a collection of indexed workflow triggers, promoting clearer understanding for future reference. * Refactor memory caching feature into separate module This commit separates the memory caching feature from the Elsa.Common module into a distinct Elsa.Caching module. This includes moving and renaming related files, such as the MemoryCacheFeature class and associated dependencies. The references in other modules and in the main solution file have been updated accordingly to include the new Elsa.Caching module. * Add distributed caching and update async methods Introduced a distributed caching feature with extensible change token signal publishing. Updated various cache-related methods to be asynchronous for improved performance and responsiveness. Also updated some workflow identity references for clarity. * Add distributed caching with MassTransit support This addition includes the implementation of a distributed caching system with MassTransit transport. The changes introduce necessary interfaces and services, new distributed caching feature along with the support for MassTransit as a transport option. Moreover, the instance management feature has been renamed to clustering feature for better clarity. * Refactor queue naming and scope of MassTransitChangeTokenSignalPublisher Queue name construction is adjusted in the RabbitMqServiceBusFeature and AzureServiceBusFeature modules for better organization and readability. MassTransitChangeTokenSignalPublisher is now a singleton service, ensuring the signal publisher can be shared across the application, increasing efficiency and performance. Also, introduced use of DistributedCacheFeature in MassTransitDistributedCacheFeature module for better modularization. * Add caching capabilities to workflow definition service This commit introduces caching to the workflow definition service, improving the performance for retrieving workflow definitions. 4 new classes have been created (`CachingWorkflowDefinitionService`, `EvictWorkflowDefinitionServiceCache`, `WorkflowDefinitionCacheManager`, and `IWorkflowDefinitionCacheManager`), and several existing classes have been updated to support caching. The caching also includes invalidation mechanisms, ensuring data consistency. * Refactor caching mechanism in workflow definition In the workflow definition module, the explicit caching functionality related to workflow definition versioning has been removed in favor of a more streamlined approach. Additionally, the manner in which services are registered has been altered. As a result, the caching now directly involves the overall workflow definition rather than individual versions, simplifying the caching logic and potentially improving the performance. * Update MassTransitBroker and enable RealTimeWorkflows and SignalRHubs The MassTransitBroker has been updated to Memory from RabbitMq. In addition, the RealTimeWorkflows and UseWorkflowsSignalRHubs features are now enabled in the code. This change will impact how the service communicates and processes real-time requests for workflow operations. * Reformat variable types in HttpFeature The reformatting involves a list of variable types in the HttpFeature module. Each type now appears on a new line for improved readability, making the code easier to maintain and review. * Update HTTP workflows cache invalidation handler XML comment * Remove unnecessary using directives Unnecessary using directives were deleted across several files in the Elsa.Http module. This simplifies the code and will possibly improve execution speed. Specific deletions include those for Encoding, Unicode, Extensions, Collections.Generic, Linq, Text, and Tasks namespaces. * Refactor HttpWorkflowsMiddleware constructor This commit simplifies the HttpWorkflowsMiddleware class constructor. It removes the intermediary variables `_next` and `_options` and directly uses the passed arguments in the constructor. Now, the `next` and `options` parameters are used directly throughout the middleware. * Simplify workflow retrieval in HttpWorkflowsMiddleware This refactoring replaces the use of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync with a single method, FindWorkflowAsync. This simplifies the middleware code and likely improves performance by reducing the number of database queries or service calls required to retrieve a workflow. * Refactor workflow retrieval in HttpBookmarkProcessor Simplified the workflow retrieval process in HttpBookmarkProcessor.cs. Replaced FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods with a single FindWorkflowAsync call. This reduces the complexity and improves efficiency in retrieving workflow. * Optimize FindWorkflowAsync method in HttpWorkflowsCacheManager Removed redundant lines of code to simplify workflow search functionality. This change simplified the FindWorkflowAsync method by directly calling the FindWorkflowAsync function in the workflowDefinitionService, thus increasing code readability and efficiency. * Refactor Endpoint.cs for workflow retrieval The method for obtaining a workflow in the Endpoint.cs script has been refactored and streamlined. The 'GetWorkflowDefinition' method is replaced by the 'GetWorkflowAsync' method which directly retrieves the workflow, without the intermediate step of materializing the workflow definition. This shortens the code and simplifies the process. * Refactor InputFunctionsDefinitionProvider constructor The constructor for InputFunctionsDefinitionProvider has been simplified by removing unnecessary private fields. Services are now directly used in the method instead of being stored in fields. This improves readability and reduces complexity in the class structure. * Refactor WorkflowInstance with improved state handling Simplified the methods for handling workflow and workflow state in the WorkflowInstance class. The refactoring also included some code clean-ups and variable renaming. The new implementation provides better readability and maintainability of the code by reducing unnecessary lines and improving structuring of objects and responses. * Remove unused IBookmarkManager and update workflow functions IBookmarkManager from ProtoActorWorkflowRuntime.cs file is removed due to its redundant status. Additionally, the "FindAsync" method has been updated to use a cancellation token. Also, annotations were added to the "ExportWorkflowStateAsync" and "ImportWorkflowStateAsync" methods to flag calls to functions that require unreferenced code. * Remove unused ReSharper directive Unused ReSharper directive in the file IndexTriggers.cs was identified and therefore removed. This change makes the code cleaner and easier to read. * Update activity invocation in workflow runtime Updated the DefaultBackgroundActivityInvoker service in the Elsa.Workflows.Runtime module to annotate the ExecuteAsync method with "RequiresUnreferencedCode" attribute. This change is made considering the potential code trimming issue. Additionally, simplified the process of fetching workflow by directly using FindWorkflowAsync method instead of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods. * Refactor code to simplify workflow definition loading The code for finding and materializing workflow definitions has been simplified. Instead of loading the definition and materializing it into a workflow in separate steps, a new method called FindWorkflowAsync has been introduced to perform both actions at once. This reduces redundancy and makes the code more readable. * Refactor WorkflowHostFactory to streamline workflow creation This commit simplifies the workflow creation process in WorkflowHostFactory. It removes redundant code and extraneous methods, specifically the overloaded CreateAsync method which used WorkflowDefinition. Now, it directly finds and uses the Workflow instance, thereby simplifying the code base and improving maintainability. * Refactor workflow retrieval in WorkflowInstance.cs Changed the way workflow instances are retrieved from the WorkflowDefinitionService. Instead of obtaining the workflow definition and then materializing the workflow from it, the workflow is directly retrieved using the FindWorkflowAsync function. This simplifies the code and avoids unnecessary null-checks. * Remove unnecessary whitespace in WorkflowInstance.cs An extraneous whitespace character was identified and removed in the WorkflowInstance.cs file. This change contributes towards maintaining clean and readable code in the Elsa.ProtoActor module. * Remove unnecessary comment in ProtoActorWorkflowRuntime The unnecessary comment ("Load the workflow definition.") in the method TryStartWorkflowAsync of the ProtoActorWorkflowRuntime.cs file was removed. This is part of an ongoing effort to keep the codebase clean and readable. * Update workflow management features and handlers Added explicit notification handlers for DeleteWorkflowInstances and RefreshActivityRegistry in WorkflowManagementFeature.cs. Also, renamed RefreshActivityRegistryHandler.cs to RefreshActivityRegistry.cs for better clarity. * Add multiple log record support to workflow execution log stores The major change of this commit is the addition of methods to add multiple log records in the WorkflowExecutionLogStore, across different storage modules such as EntityFramework, MongoDB, Dapper, Elasticsearch and Memory. This ensures consistency and uniform behavior across different storage types. Furthermore, some reformatting and tidying up of the code were undertaken to maintain readability and clarity. * Remove redundant workflow definition check The workflow definition existence check and related service retrieval were removed from HttpWorkflowsMiddleware.cs. It was determined that this check was unnecessary as the workflow definition's existence is guaranteed at this point in the process, reducing redundancy in the code. * Improve cancellation token usage in workflow execution This commit refines the usage of cancellation tokens during the execution of workflows in Elsa.Server and HttpWorkflowsMiddleware. Previously, a cancellation token pair was created before ExecuteWithinTimeoutAsync was called, which limited duration control solely to that method. Now, cancellation tokens are included within ExecuteWithinTimeoutAsync method. This allows the method to observe any cancellation initiated by outer scopes, enhancing control over the timeout of operations. * Add PersistStateAsync method to WorkflowHost A new PersistStateAsync method has been added to the WorkflowHost, which enables the host to directly persist its own state. The method has been integrated into the DefaultWorkflowRuntime and HttpWorkflowsMiddleware. This update eliminates the need to continuously get instances of IWorkflowInstanceManager to save state, which improves efficiency and code readability. * Refactor DefaultAlterationRunner service This update simplifies the DefaultAlterationRunner service, reducing the number of code lines and removing unnecessary references. The workflow materialization step has been merged with the find workflow step, and unused namespaces have been dropped. * Refine wording in IWorkflowHost interface documentation The documentation for the 'CanStartWorkflowAsync' method in the IWorkflowHost interface has been cleaned up. The superfluous "or not" verbiage has been removed, making it easier to understand the method's function. * Remove Redis from DistributedCachingTransport The Redis option was removed from the DistributedCachingTransport enumeration. This transport isn't currently implemented. * Remove 'useDistributedCaching' constant The 'useDistributedCaching' constant was removed from `Program.cs`, and conditional logic was updated to use `distributedCachingTransport != DistributedCachingTransport.None`. A new option 'None' was added to the `DistributedCachingTransport` enum to facilitate this change. * Update package tags in MassTransit project file The package tags in the Elsa.Caching.Distributed.MassTransit project file was updated to consolidate the tags, changing 'mass-transit' to 'masstransit'. This change better aligns with standard naming conventions and improves searchability. * Refactor distributed caching implementation This commit involves an extensive refactor of the distributed caching implementation. Distributed caching related code and resources were moved into an independent 'Elsa.Caching.Distributed' module. The interface 'IDistributedChangeTokenSignaler' was deleted and its functionality was replaced by 'IChangeTokenSignalInvoker'. * Refactor order of parameters in GetOrCreateAsync method The order of parameters in the GetOrCreateAsync method within the CachingWorkflowDefinitionStore class has been changed. This change ensures that the `key` parameter is now first, followed by the `factory` parameter. This improves code readability and aligns with standard coding practices. * Refactor cache retrieval in Workflow service Refactoring was done to streamline the way objects are retrieved from cache in the Workflow service. Duplicated code was condensed into a new `GetFromCacheAsync` method, which is now called in the existing methods, thus increasing maintainability and reducing the possibility of errors. * Update method descriptions and fix comments formatting Method descriptions in various contracts have been updated to more accurately reflect their function regarding record addition and updating in the persistence store. All double comment markers (/// ///) have also been corrected to the standard (///) across multiple classes. * Remove unused caching methods in ModuleExtensions The commit removes the unused methods, `UseMemoryCache` and `UseDistributedCache` from the `ModuleExtensions.cs` file. The removal is part of a wider cleanup and refactoring effort to streamline the codebase and improve legibility. * Remove redundant PrimaryKeyName in DapperWorkflowExecutionLogStore The "PrimaryKeyName" constant was removed in DapperWorkflowExecutionLogStore. This change simplifies the initialization of the '_store' property, reducing unnecessary redundancy and complexity. The refactored code maintains the same functionality but improves readability and maintainability. * Refactor SaveAsync methods in Elsa.Dapper Store The SaveAsync functions have been updated in the Store.cs file inside the Elsa.Dapper module. They now include cancellation token parameters and specify that they add or update records, providing clearer distinction and flexibility. * Refactor store initialization in Elsa.Dapper modules Removed the redundant usage of primary keys during the store initialization across Elsa.Dapper module. Simplified the SaveAsync methods by removing the parameter for primary key, making the code cleaner and more maintainable. This refactoring does not affect the module's functionality. * Refactor UserStore in Elsa.Dapper module The code was adjusted to improve readability within the Elsa.Dapper module's UserStore. Two lines that were previously combined have now been separated into distinct lines, making the code structure more clear. * Refactor constructor arguments in MongoDb module Simplified several classes in the MongoDb module by injecting dependencies directly through the constructor instead of assigning them to private readonly fields. This improves readability and removes unnecessary code lines. Also added JetBrains.Annotations where applicable. * Fix comment syntax in IWorkflowInstanceStore A syntax error in the comments for the method SaveManyAsync (in IWorkflowInstanceStore interface) has been corrected. This change ensures that the remarks section of the method is properly formatted and correctly displayed in documentation. * Remove ComputeBookmarkHash from IHttpWorkflowsCacheManager The ComputeBookmarkHash method was removed from IHttpWorkflowsCacheManager to declutter the interface. The functionality was moved and adapted in the HttpWorkflowsMiddleware class to maintain the original functionality. * Add logging to HttpWorkflowsMiddleware In this update, the HttpWorkflowsMiddleware class has been modified to include logging. Specifically, warning logs have been added to track workflow-related processes and to notify if mentioned bookmarks or workflow instances are not found. * Update consumer configuration in MassTransitFeature This commit modifies the consumer configuration in the MassTransitFeature. Instead of hardcoding the consumer type to DispatchCancelWorkflowsRequestConsumer, it now uses the dynamic consumer type retrieved from the context, making the feature more adaptable for different scenarios. * Change default MassTransitBroker to Memory The default value for the variable useMassTransitBroker in Elsa.Server.Web's Program.cs file has been modified. It has been changed from RabbitMq to Memory to change the message broker used by MassTransit in the application. * Remove Datadog.Trace package from Directory.Packages.props The Datadog.Trace package with version 2.49.0 has been removed from the Directory.Packages.props file. This change reflects the fact that this package is no longer required in our project.
2024-04-10 09:51:40 +00:00
Fix WorkflowActivity to Use Cached Workflow Definitions for Consistent Behavior (#5223) * Replace IServiceScopeFactory with IServiceProvider in WorkflowRunner Unused dependencies were removed from the workflow runner service. The IServiceScopeFactory was replaced with IServiceProvider to better handle the creation and deletion of service scopes, resulting in cleaner code with less manual scope management. Microsoft.Extensions.DependencyInjection and System.Diagnostics.CodeAnalysis were removed as they were no longer necessary. * Refactor WorkflowDefinitionActivity to use WorkflowDefinitionService The WorkflowDefinitionActivity class has been refactored to make use of the WorkflowDefinitionService instead of the WorkflowDefinitionStore. This fixes #5222 by ensuring the same activity instances are used in the graph model of the workflow execution context. * Add workflow filtering and caching functionality Added methods to `WorkflowDefinitionService` to find workflow definitions and workflows using filter criteria. A key generation method for caching filtered workflows was also added to `WorkflowDefinitionCacheManager`. The implementation includes generating a hash of the filter parameters and using this hash as a cache key, providing efficient caching functionality for filtered searches. * Refactor TriggerIndexer to handle only ITrigger activities The code in TriggerIndexer has been refactored to deal specifically with ITrigger activities, streamlining its behavior. Removed code related to handling non-ITrigger activities and simplified the workflow creation process. The extraction of "startable" nodes now directly filters and casts to ITrigger, reducing complexity and increasing readability. * Update caching service to support filter-based search The CachingWorkflowDefinitionService has been updated to support workflow definition and workflow search based on filter criteria. The update also includes change of class scope from public to internal. Further, it resolves the missing reference by switching from Elsa.Caching.Contracts to Elsa.Caching. * Optimize Elsa project imports and use explicit cache variable names This commit removes superfluous import references, relocates the 'IChangeTokenSignaler' contract into the 'Elsa.Caching' namespace, and replaces ambiguous 'cache' variable names with more explicit 'memoryCache' across several files. Additionally, new package references have been added and access modifiers have been changed to improve encapsulation. Cleanup enhances readability and maintainability of the codebase. * Add .DotSettings file to Elsa.Caching module A new .DotSettings file has been added to the Elsa.Caching module. This file is used for namespace configuration, specifically to skip the "contracts" folder in code inspections. * Update workflow interfaces to support filter queries The update extends `IWorkflowDefinitionCacheManager` and `IWorkflowDefinitionService` interfaces. Functions are added to allow creating filter cache keys and finding workflow definitions and workflows using a new `WorkflowDefinitionFilter`. This enhances querying flexibility by enabling filtered searches. * Add WorkflowDefinitionVersionId to WorkflowTriggerEqualityComparer A new property, WorkflowDefinitionVersionId, has been added to the object being serialized in WorkflowTriggerEqualityComparer. This change allows for a more accurate comparison between workflow triggers, considering not just the workflow definition ID but also its version. * Update service registration types in WorkflowsFeature Changed the registration type for both IHasher and IBookmarkHasher services from Scoped to Singleton in the workflows feature configuration. This alteration aims to improve application performance and manage service lifetimes more efficiently. * Remove Open.Linq.AsyncExtensions dependency The Open.Linq.AsyncExtensions package reference was removed across the project. The usage within the CachingWorkflowDefinitionStore was updated accordingly to maintain functionality. * Move System.Linq.Dynamic.Core package reference The System.Linq.Dynamic.Core package reference was moved from the Directory.Build.props file to the Elsa.Workflows.Management.csproj file. This change reflects the specific dependency of the Elsa.Workflows.Management module on System.Linq.Dynamic.Core, without impacting other modules. * Implement caching for HTTP workflows This update introduces caching mechanisms for HTTP workflows, which significantly improves their performance. The changes involve creating a `CacheManager` and `CachingHttpWorkflowLookupService`, and modifying some existing components to use the new caching mechanism. Additionally, the `HttpWorkflowsCacheManager` was renamed to `HttpWorkflowsCacheInvalidationManager` to better reflect its role. * Refactor cache management across modules This commit refactor the cache management across various modules. The 'ICacheManager' interface now includes methods for triggering and getting change tokens, and the 'HttpWorkflowsCacheInvalidationManager' has been renamed to 'HttpWorkflowsCacheManager'. The caching functionality in 'WorkflowDefinitionService' and other similar services have been updated to use these new methods, improving consistency and maintainability. * Enable caching in Elsa.Server.Web The "useCaching" variable has been set to true to enable caching. Simultaneously, the method name "UseCachingStores" has been refactored to "UseCache". Conditional statements have been added to check the "useCaching" variable before invoking caching. * Rename method UseCaching to UseCache In the Elsa.Server.Web and Elsa.Http project files, the method UseCaching has been renamed to UseCache. This modification is aimed at bridging naming inconsistencies and maintaining naming standards across the application. * Update HttpCacheFeature class description The class summary for HttpCacheFeature has been revised. Originally, it stated that the class was used for installing services related to HTTP services and activities, but it actually focuses more on HTTP caching. * Remove unused Configure method from HttpCacheFeature The Configure method in HttpCacheFeature was found to be redundant as it wasn't doing any significant work or contributing to any functionality. It has therefore been removed to clean up the code and avoid confusion. * Add 'bug/*' to workflow triggers This commit includes 'bug/*' to the list of triggers in our GitHub Actions workflow. Now, any push or pull request under a 'bug/*' branch will trigger the workflow.
2024-04-15 08:06:27 +00:00
if (useCaching)
management.UseCache();
Add Component Testing Framework (#5261) * Add application component tests Multiple new test files were added to deliver application component tests. This move improves testing by adding integration tests that cover overall system behavior and checking end-to-end actions. Ensuring the system functions correctly as a whole. In the process, updating some package versions to maintain compatibility. * Add RefitSettings helper and revise API client service configuration The commit introduces a 'RefitSettingsHelper' for Elsa API client and revises the way the API client services are configured. It also makes improvements to the WorkflowServerTestWebAppFactory for component testing. Some endpoint contracts related to workflow execution are also updated to have optional parameters. * Remove old tests and add new workflow tests This commit removes old, unnecessary tests and incorporates new workflow tests. It also improves the Elsa API client JSON serializer and adds a helper for HttpResponseMessage. Lastly, the commit introduces changes to properly configure the test logging and to manage application settings. * Add HttpHelloWorld workflow tests A new component test scenario, HttpHelloWorldTests, has been created for testing an HttpHelloWorld workflow. This involves asserting if a workflow responds correctly with "Hello World". Furthermore, an HTTP workflow client has been introduced in the WorkflowServerTestWebAppFactory class to provide a base address for workflow API calls. * Add new test file and update workflow execution tests This change adds a new test file "fork-1.json" to the Elsa.Workflows.Api.ComponentTests project. Also, updates were made throughout the tests to replace the WorkflowServerTestWebAppFactory with a fixture, allowing the tests to run in parallel. Lastly, unnecessary warning suppression was removed from the Elsa.Workflows.Core extension method. * Add filter for .json and .elsa files in BlobStorageWorkflowProvider This change adds a BrowseFilter in the BlobStorageWorkflowProvider options. This filter checks for files that end with .json or .elsa and includes only these files when browsing through the blob storage. This filter helps prioritize specific workflow file types. * Rename WorkflowServerTestWebAppFactoryFixture and update usage The old class name "WorkflowServerTestWebAppFactoryFixture" has been replaced with the more accurate "WorkflowServerWebAppFactoryFixture". All references to the previous name in other classes were also updated accordingly. In addition, the directory key in the method "CreateConvoyOptionsBuilder" has been updated from "Workflows" to "Scenarios". * Update test fixture in workflow tests The commit updates the test fixture in two test classes: HttpHelloWorldTests and HelloWorldTests. The former test fixture, WorkflowServerTestWebAppFactoryFixture, was replaced by WorkflowServerWebAppFactoryFixture to accurately match the testing needs. * Update .csproj file paths and reorganize tests The commit modifies the file paths for several test scenario files in the Elsa.Workflows.Api.ComponentTests.csproj, reflecting a reorganization of the tests. Previously static paths have been updated to new paths under 'Scenarios'. Additionally, two new test files related to 'LogPersistenceModes' have been included in the project. * Add tests for log persistence modes This commit introduces two new test scenarios for logging persistence modes and includes a related test called 'HelloWorldWorkflow'. These tests cover scenarios where certain workflow inputs should be stored and others shouldn't, thereby testing the log persistence feature. This ensures that the logging behavior respects the specified persistence mode. * Add log persistence tests and update LogPersistenceMode enum The commit contains the addition of new log persistence tests for verifying correctness of log persistence behavior. Furthermore, the LogPersistenceMode enum has been updated, replacing 'Default' with 'Inherit'. This change makes the mode's purpose clearer. Lastly, new test scenarios and test data files were added for more comprehensive testing. * Remove obsolete component tests and support files The files removed are no longer necessary for the current state of the application. They include various component tests and their related support files within the Elsa.Workflows.Api.ComponentTests project. By removing these, the project structure is cleaner and only contains relevant tests. * Add dispatch workflow scenario tests and necessary helper classes This commit includes two new tests for dispatching workflows, along with the creation of new 'ChildWorkflow' and 'DispatchAndWaitWorkflow' classes. Auxiliary helpers and services have been added to aid in managing workflow events and signals for these tests. The 'ComponentTest' has also been upgraded to support disposal handling. * Remove ITestOutputHelper dependency from test classes Removed the dependency on ITestOutputHelper in multiple test classes across various workflow scenarios. This change simplifies the test class constructors by reducing the number of required dependencies, contributing to cleaner and leaner code. * Add 'Hello World' scenario to WorkflowCompletion tests The 'Hello World' scenario was moved into WorkflowCompletion tests, along with changes in workflow definition identifiers. As part of these changes, the 'hello-world.json' file was updated; a new file under the same name was created in the WorkflowCompletion area and the workflow identifiers in basic and workflow completion tests were updated accordingly. Additionally, 'fork-1.json' has been renamed to 'fork.json'. * Add support for cluster hosting tests This commit introduces a suite of integration tests designed to validate the behaviour of hosting multiple instances of Elsa in a clustered environment. These tests simulate a typical clustered hosting scenario by using 'App', 'Cluster', and 'Infrastructure' objects to emulate different instances of the Elsa workflow engine running on separate servers. Name changes were made to certain classes and methods to reflect their new scopes and roles within the testing environment. * Add performance tests and improve component tests Added a new performance tests project scaffold, complete with its own project file, build properties file, and a dummy test. Updated component tests to improve multi-pod testing, primarily through the addition of additional service scopes and asserting activity registry synchronization. These changes also required updates to existing project and props files as well as the solution file. * Update ActivityRegistrySyncTests and Infrastructure Added a reference to Services in ActivityRegistrySyncTests and removed unnecessary whitespace in both files. The test component Elsa.Workflows has been modified to import newly added services, ensuring all tests are running with the expected resources and services. * Fix comment * Add NOOP implementations for stores * Update PostgreSQL image and adjust test timings The PostgreSQL image used for testing has been updated to the latest version from 13.3-alpine. Timeouts in ISignalManager and DispatchWorkflowsTests have been reduced for efficiency. A delay in the ChildWorkflow has also been decreased. Additionally, an 'ImportWorkflowActivity' test in ActivityRegistrySyncTests has been marked as not yet implemented.
2024-04-26 13:49:14 +00:00
management.SetDefaultLogPersistenceMode(LogPersistenceMode.Inherit);
management.UseReadOnlyMode(useReadOnlyMode);
})
.UseWorkflowRuntime(runtime =>
{
if (useMongoDb)
runtime.UseMongoDb();
else if (useDapper)
runtime.UseDapper();
else
runtime.UseEntityFrameworkCore(ef =>
{
if (useSqlServer)
ef.UseSqlServer(sqlServerConnectionString!);
else if (usePostgres)
ef.UsePostgreSql(postgresConnectionString!);
else if (useCockroachDb)
ef.UsePostgreSql(cockroachDbConnectionString!);
else
ef.UseSqlite(sqliteConnectionString);
ef.RunMigrations = runEFCoreMigrations;
});
if (useProtoActor)
{
runtime.UseProtoActor(proto => proto.PersistenceProvider = _ =>
{
if (useSqlServer)
return new SqlServerProvider(sqlServerConnectionString!, true, "", "proto_actor");
else
return new SqliteProvider(new SqliteConnectionStringBuilder(sqliteConnectionString));
});
}
if (useMassTransit)
runtime.UseMassTransitDispatcher();
runtime.WorkflowInboxCleanupOptions = options => configuration.GetSection("Runtime:WorkflowInboxCleanup").Bind(options);
Implement dispatch channels (#4949) * Add DispatchWorkflowOptions and update MassTransit configuration Introduced a new class `DispatchWorkflowOptions` to provide workflow dispatch options. Updated MassTransit configuration to include NET6.0 and NET7.0 support, and ensure correct MassTransit version usage per target framework version. * Add support for configurable MassTransit message dispatching Implemented a feature which allows for configurable MassTransit message dispatching. Added support for specifying channels and message brokers. Message dispatching code was massively refactored and relevant endpoints and response models were updated to support new message dispatching features. * Add IEndpointChannelFormatter interface and implementation An interface for formatting channel queue names, 'IEndpointChannelFormatter', has been added, along with its default implementation 'DefaultEndpointChannelFormatter'. The code that uses hardcoded queue name formatting has been modified to use the new formatter instead, making it more configurable and reusable. The implementation of the formatter uses the 'Humanizer' library to kebab-case the channel names. * Add channel dispatch option to workflow activities Introduced `WorkflowDispatcherChannelOptionsProvider` to provide dropdown channel options for workflow dispatch-related activities. Updated `DispatchWorkflow` and `BulkDispatchWorkflows` activities to include a new dropdown input for specifying a dispatch channel. Also, included the selected channel name in the `DispatchWorkflowOptions` during workflow dispatching process. * Update MassTransit configurations and remove unused code The MassTransit setup in Elsa.MassTransit module has been simplified by removing conditional code for different .NET versions. Additionally, unused parameter '__X_Channel' in 'DispatchWorkflowDefinition' was removed. Lastly, the MassTransit broker in Elsa.Server.Web was switched from RabbitMq to AzureServiceBus. * Refactor dispatch workflow classes and methods Simplified the class, method and variable names related to workflow dispatching in the Elsa.Workflows.Runtime module. For example, the 'WorkflowDispatcherChannelDescriptor' class was renamed to 'DispatcherChannel'. This refactoring was performed to make code more readable and maintainable by removing redundant wording in the naming convention. * Update GitHub Workflow to support feature and issue branches The workflow changes add support for feature and issue branches. Now, it extracts the branch name and verifies the commit exists in the given branch rather than just 'main'. The versioning scheme is also modified to include the branch name and not just the run number. * Add 'bug/*' to triggering branches in packages workflow The 'bug/*' pattern was missing from the triggers that initiate the GitHub actions within our packages workflow. This update includes any branch with a 'bug/' prefix to the list, allowing bug-related branches to start jobs in our CI/CD pipeline. * Remove 'issue/*' and 'bug/*' branches from packages workflow The 'issue/*' and 'bug/*' branches have been removed from the GitHub action workflow for packages. This change was made to simplify the workflow and optimize the triggering of package building. * Update GitHub workflow to handle main branch versioning This commit modifies the GitHub workflow script to accommodate changes when the branch name is "main." If the branch name is "main", a preview version is used. It also updates script execution to print the branch name for easier debugging and verifies commit existence on the correct branch instead of dispatch channels. * Enclose branch names in quotes in packages.yml The update modifies the branch names in the packages.yml GitHub Actions workflow file. The change consists of enclosing the branch names 'main' and 'feature/*' in single quotes, ensuring compatibility and preventing potential string interpretation issues. * Update GitHub workflows package configuration The workflows package configuration has been updated to specifically watch for changes on 'feature/dispatch-channels' rather than on all feature branches. This change will prevent unnecessary builds on less relevant feature branches. * Update trigger branches in packages workflow The triggering branches in the packages workflow have been updated. Previously, only changes in the 'main' and 'feature/dispatch-channels' would trigger the workflow, now any 'feature/*' branch will. This will cause more frequent and comprehensive testing. * Add 'patch/*' to workflow trigger branches This update adds 'patch/*' to the list of branches in .github/workflows/packages.yml that can trigger the workflow. It will allow the workflow to be initiated not just for main and feature branches, but also for patches. * Add 'preview/*' to workflow triggers This commit adds a new trigger for the GitHub Actions workflow. It now also responds to push events on 'preview/*' branches, allowing for automated testing and building of these preview branches. * Update branch name extraction in GitHub Actions The extraction of the branch name has been slightly modified in the packages.yml GitHub workflow file. This alteration ensures the correct branch name is obtained for further processing within the workflow without any discrepancy. * Update branch name extraction in packages.yml Corrected the syntax for extracting the branch name within the packages.yml github workflow file. Added an extra line to print out the ref which might be useful for debugging. * Update branch name extraction in GitHub workflow The commit simplifies the way the branch name is being extracted from the GitHub ref in the packages.yml workflow file. The new method employs straightforward string manipulation, making it easier to understand and debug in case of potential issues. * Add extraction of branch name in workflow Added a new line in the GitHub workflow file (.github/workflows/packages.yml) to extract the last part after the final slash from the branch name. This enhancement allows cleaner naming conventions, especially in cases where branches are named feature/issue-123, as it will only retain 'issue-123'. * Update package naming in Github workflow The Github workflow has been updated to handle package naming more effectively. Previously, the branch name was used directly for package versioning. Now, the last part of the branch name is extracted and used as the package prefix. If the branch name is "main", the package prefix is set to "preview". * Move and add environment variable assignments The placement of the assignment for BRANCH_NAME environment variable was moved for better readability. Additionally, the PACKAGE_PREFIX environment variable was also added. These environment variables are crucial for subsequent steps in the GitHub workflow. * Add workflow dispatch validation and response handling Removed several specific dispatch response classes and consolidated all types of dispatch responses into a single DispatchWorkflowResponse class. Added a new ValidatingWorkflowDispatcher service to validate dispatch requests before they're sent. Updated several classes to work with these changes, including the BackgroundWorkflowDispatcher, MassTransitWorkflowDispatcher, and the API endpoint class. * Handle dispatch workflow failures with exceptions The DispatchWorkflow and BulkDispatchWorkflows activities now throw a FaultException when the dispatch operations fail. Previously, these operations were not checking for success and could fail silently. Now, an unsuccessful dispatch response results in a FaultException with an error message from the dispatch response.
2024-02-16 09:56:30 +00:00
runtime.WorkflowDispatcherOptions = options => configuration.GetSection("Runtime:WorkflowDispatcher").Bind(options);
if (useMemoryStores)
{
runtime.ActivityExecutionLogStore = sp => sp.GetRequiredService<MemoryActivityExecutionStore>();
runtime.WorkflowExecutionLogStore = sp => sp.GetRequiredService<MemoryWorkflowExecutionLogStore>();
runtime.WorkflowInboxStore = sp => sp.GetRequiredService<MemoryWorkflowInboxMessageStore>();
}
Fix WorkflowActivity to Use Cached Workflow Definitions for Consistent Behavior (#5223) * Replace IServiceScopeFactory with IServiceProvider in WorkflowRunner Unused dependencies were removed from the workflow runner service. The IServiceScopeFactory was replaced with IServiceProvider to better handle the creation and deletion of service scopes, resulting in cleaner code with less manual scope management. Microsoft.Extensions.DependencyInjection and System.Diagnostics.CodeAnalysis were removed as they were no longer necessary. * Refactor WorkflowDefinitionActivity to use WorkflowDefinitionService The WorkflowDefinitionActivity class has been refactored to make use of the WorkflowDefinitionService instead of the WorkflowDefinitionStore. This fixes #5222 by ensuring the same activity instances are used in the graph model of the workflow execution context. * Add workflow filtering and caching functionality Added methods to `WorkflowDefinitionService` to find workflow definitions and workflows using filter criteria. A key generation method for caching filtered workflows was also added to `WorkflowDefinitionCacheManager`. The implementation includes generating a hash of the filter parameters and using this hash as a cache key, providing efficient caching functionality for filtered searches. * Refactor TriggerIndexer to handle only ITrigger activities The code in TriggerIndexer has been refactored to deal specifically with ITrigger activities, streamlining its behavior. Removed code related to handling non-ITrigger activities and simplified the workflow creation process. The extraction of "startable" nodes now directly filters and casts to ITrigger, reducing complexity and increasing readability. * Update caching service to support filter-based search The CachingWorkflowDefinitionService has been updated to support workflow definition and workflow search based on filter criteria. The update also includes change of class scope from public to internal. Further, it resolves the missing reference by switching from Elsa.Caching.Contracts to Elsa.Caching. * Optimize Elsa project imports and use explicit cache variable names This commit removes superfluous import references, relocates the 'IChangeTokenSignaler' contract into the 'Elsa.Caching' namespace, and replaces ambiguous 'cache' variable names with more explicit 'memoryCache' across several files. Additionally, new package references have been added and access modifiers have been changed to improve encapsulation. Cleanup enhances readability and maintainability of the codebase. * Add .DotSettings file to Elsa.Caching module A new .DotSettings file has been added to the Elsa.Caching module. This file is used for namespace configuration, specifically to skip the "contracts" folder in code inspections. * Update workflow interfaces to support filter queries The update extends `IWorkflowDefinitionCacheManager` and `IWorkflowDefinitionService` interfaces. Functions are added to allow creating filter cache keys and finding workflow definitions and workflows using a new `WorkflowDefinitionFilter`. This enhances querying flexibility by enabling filtered searches. * Add WorkflowDefinitionVersionId to WorkflowTriggerEqualityComparer A new property, WorkflowDefinitionVersionId, has been added to the object being serialized in WorkflowTriggerEqualityComparer. This change allows for a more accurate comparison between workflow triggers, considering not just the workflow definition ID but also its version. * Update service registration types in WorkflowsFeature Changed the registration type for both IHasher and IBookmarkHasher services from Scoped to Singleton in the workflows feature configuration. This alteration aims to improve application performance and manage service lifetimes more efficiently. * Remove Open.Linq.AsyncExtensions dependency The Open.Linq.AsyncExtensions package reference was removed across the project. The usage within the CachingWorkflowDefinitionStore was updated accordingly to maintain functionality. * Move System.Linq.Dynamic.Core package reference The System.Linq.Dynamic.Core package reference was moved from the Directory.Build.props file to the Elsa.Workflows.Management.csproj file. This change reflects the specific dependency of the Elsa.Workflows.Management module on System.Linq.Dynamic.Core, without impacting other modules. * Implement caching for HTTP workflows This update introduces caching mechanisms for HTTP workflows, which significantly improves their performance. The changes involve creating a `CacheManager` and `CachingHttpWorkflowLookupService`, and modifying some existing components to use the new caching mechanism. Additionally, the `HttpWorkflowsCacheManager` was renamed to `HttpWorkflowsCacheInvalidationManager` to better reflect its role. * Refactor cache management across modules This commit refactor the cache management across various modules. The 'ICacheManager' interface now includes methods for triggering and getting change tokens, and the 'HttpWorkflowsCacheInvalidationManager' has been renamed to 'HttpWorkflowsCacheManager'. The caching functionality in 'WorkflowDefinitionService' and other similar services have been updated to use these new methods, improving consistency and maintainability. * Enable caching in Elsa.Server.Web The "useCaching" variable has been set to true to enable caching. Simultaneously, the method name "UseCachingStores" has been refactored to "UseCache". Conditional statements have been added to check the "useCaching" variable before invoking caching. * Rename method UseCaching to UseCache In the Elsa.Server.Web and Elsa.Http project files, the method UseCaching has been renamed to UseCache. This modification is aimed at bridging naming inconsistencies and maintaining naming standards across the application. * Update HttpCacheFeature class description The class summary for HttpCacheFeature has been revised. Originally, it stated that the class was used for installing services related to HTTP services and activities, but it actually focuses more on HTTP caching. * Remove unused Configure method from HttpCacheFeature The Configure method in HttpCacheFeature was found to be redundant as it wasn't doing any significant work or contributing to any functionality. It has therefore been removed to clean up the code and avoid confusion. * Add 'bug/*' to workflow triggers This commit includes 'bug/*' to the list of triggers in our GitHub Actions workflow. Now, any push or pull request under a 'bug/*' branch will trigger the workflow.
2024-04-15 08:06:27 +00:00
if (useCaching)
runtime.UseCache();
Add caching to workflow runtime and workflow management stores (#5174) * Add caching to workflow runtime and workflow management stores The update introduces caching to workflow runtime and workflow management stores to enhance performance. This is achieved by adding decorators for several stores, which cache records to reduce database fetches. Additionally, a signaler for change tokens allows for cache invalidation when changes occur. The MemoryCache feature has also been updated to include Scrutor for decoration and the caching duration can be configured through the new CachingOptions class. * Refactor WorkflowsMiddleware for HTTP Endpoint bookmarks and triggers The WorkflowsMiddleware has been extensively refactored to handle HTTP Endpoint bookmarks and triggers. This involves breaking down the InvokeAsync method by extracting parts of its functionality into separate helper methods such as FindTriggersAsync and FindBookmarksAsync. Moreover, Assist with authorization checks, workflow execution within request timeout, and handling of workflow faults has been improved to be more efficient and clearly segmented. * Update HTTP endpoint authorization to use Workflow context The authorization process in the AuthenticationBasedHttpEndpointAuthorizationHandler class has been updated to use the Workflow context instead of the WorkflowInstanceId string. The AuthorizeHttpEndpointContext model has been correspondingly changed to include a Workflow property, thereby strengthening the link between authorization and specific workflows. * Add FindAsync methods to trigger and bookmark stores The code adjustments add new FindAsync methods to the trigger and bookmark store contracts as well as all their concrete implementations (MongoDb, Memory and EFCore). These methods support fetching the first record matching a given filter. The adjustments also include minor syntax improvements and the addition of [UsedImplicitly] attributes where needed. * Refactor WorkflowsMiddleware for improved workflow handling The code was refactored to simplify the flow of handling workflows in the WorkflowsMiddleware class. Specifically, the methods to start and resume a workflow have been extracted to improve code readability. Further, handleErrorMiddlewares was also updated to better manage instances where no valid workflows or base paths are found. * Add caching functionality to WorkflowsMiddleware Added IMemoryCache usage in the WorkflowsMiddleware to cache lookup results for workflows and their associated triggers. This will reduce the number of database operations required when searching for workflows, thus improving performance. The cache is maintained for one minute before it is refreshed. * Implement dynamic cache duration for workflows The code has been updated to have a dynamic cache duration for the workflows instead of a hardcoded one minute. By using the CachingOptions service, the cache duration can now be set in the configuration making it more flexible and adaptable to different performance needs. * Add HttpWorkflowsCacheManager for caching HTTP workflows This commit includes the implementation of IHttpWorkflowsCacheManager for caching of HTTP workflows. New handlers have been added to invalidate cache on workflow updates. Additionally, WorkflowsMiddleware has been renamed to HttpWorkflowsMiddleware. * Refactor workflow trigger handling and caching The refactoring includes deletion of `IndexWorkflowTriggersHandler.cs` and creation of `IndexTriggers.cs` thus revising the workflow trigger indexing approach. Also, revamped the `ITriggerIndexer` interface which now handles deletion of triggers with specific workflow and filter. Furthermore, the caching mechanism in `HttpWorkflowsCacheManager.cs` is modified to handle eviction of workflow definitions and triggers separately boosting its efficiency. * Add summary to IndexedWorkflowTriggers A summary has been added to the 'IndexedWorkflowTriggers' class, providing a brief description. This description outlines that it represents a collection of indexed workflow triggers, promoting clearer understanding for future reference. * Refactor memory caching feature into separate module This commit separates the memory caching feature from the Elsa.Common module into a distinct Elsa.Caching module. This includes moving and renaming related files, such as the MemoryCacheFeature class and associated dependencies. The references in other modules and in the main solution file have been updated accordingly to include the new Elsa.Caching module. * Add distributed caching and update async methods Introduced a distributed caching feature with extensible change token signal publishing. Updated various cache-related methods to be asynchronous for improved performance and responsiveness. Also updated some workflow identity references for clarity. * Add distributed caching with MassTransit support This addition includes the implementation of a distributed caching system with MassTransit transport. The changes introduce necessary interfaces and services, new distributed caching feature along with the support for MassTransit as a transport option. Moreover, the instance management feature has been renamed to clustering feature for better clarity. * Refactor queue naming and scope of MassTransitChangeTokenSignalPublisher Queue name construction is adjusted in the RabbitMqServiceBusFeature and AzureServiceBusFeature modules for better organization and readability. MassTransitChangeTokenSignalPublisher is now a singleton service, ensuring the signal publisher can be shared across the application, increasing efficiency and performance. Also, introduced use of DistributedCacheFeature in MassTransitDistributedCacheFeature module for better modularization. * Add caching capabilities to workflow definition service This commit introduces caching to the workflow definition service, improving the performance for retrieving workflow definitions. 4 new classes have been created (`CachingWorkflowDefinitionService`, `EvictWorkflowDefinitionServiceCache`, `WorkflowDefinitionCacheManager`, and `IWorkflowDefinitionCacheManager`), and several existing classes have been updated to support caching. The caching also includes invalidation mechanisms, ensuring data consistency. * Refactor caching mechanism in workflow definition In the workflow definition module, the explicit caching functionality related to workflow definition versioning has been removed in favor of a more streamlined approach. Additionally, the manner in which services are registered has been altered. As a result, the caching now directly involves the overall workflow definition rather than individual versions, simplifying the caching logic and potentially improving the performance. * Update MassTransitBroker and enable RealTimeWorkflows and SignalRHubs The MassTransitBroker has been updated to Memory from RabbitMq. In addition, the RealTimeWorkflows and UseWorkflowsSignalRHubs features are now enabled in the code. This change will impact how the service communicates and processes real-time requests for workflow operations. * Reformat variable types in HttpFeature The reformatting involves a list of variable types in the HttpFeature module. Each type now appears on a new line for improved readability, making the code easier to maintain and review. * Update HTTP workflows cache invalidation handler XML comment * Remove unnecessary using directives Unnecessary using directives were deleted across several files in the Elsa.Http module. This simplifies the code and will possibly improve execution speed. Specific deletions include those for Encoding, Unicode, Extensions, Collections.Generic, Linq, Text, and Tasks namespaces. * Refactor HttpWorkflowsMiddleware constructor This commit simplifies the HttpWorkflowsMiddleware class constructor. It removes the intermediary variables `_next` and `_options` and directly uses the passed arguments in the constructor. Now, the `next` and `options` parameters are used directly throughout the middleware. * Simplify workflow retrieval in HttpWorkflowsMiddleware This refactoring replaces the use of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync with a single method, FindWorkflowAsync. This simplifies the middleware code and likely improves performance by reducing the number of database queries or service calls required to retrieve a workflow. * Refactor workflow retrieval in HttpBookmarkProcessor Simplified the workflow retrieval process in HttpBookmarkProcessor.cs. Replaced FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods with a single FindWorkflowAsync call. This reduces the complexity and improves efficiency in retrieving workflow. * Optimize FindWorkflowAsync method in HttpWorkflowsCacheManager Removed redundant lines of code to simplify workflow search functionality. This change simplified the FindWorkflowAsync method by directly calling the FindWorkflowAsync function in the workflowDefinitionService, thus increasing code readability and efficiency. * Refactor Endpoint.cs for workflow retrieval The method for obtaining a workflow in the Endpoint.cs script has been refactored and streamlined. The 'GetWorkflowDefinition' method is replaced by the 'GetWorkflowAsync' method which directly retrieves the workflow, without the intermediate step of materializing the workflow definition. This shortens the code and simplifies the process. * Refactor InputFunctionsDefinitionProvider constructor The constructor for InputFunctionsDefinitionProvider has been simplified by removing unnecessary private fields. Services are now directly used in the method instead of being stored in fields. This improves readability and reduces complexity in the class structure. * Refactor WorkflowInstance with improved state handling Simplified the methods for handling workflow and workflow state in the WorkflowInstance class. The refactoring also included some code clean-ups and variable renaming. The new implementation provides better readability and maintainability of the code by reducing unnecessary lines and improving structuring of objects and responses. * Remove unused IBookmarkManager and update workflow functions IBookmarkManager from ProtoActorWorkflowRuntime.cs file is removed due to its redundant status. Additionally, the "FindAsync" method has been updated to use a cancellation token. Also, annotations were added to the "ExportWorkflowStateAsync" and "ImportWorkflowStateAsync" methods to flag calls to functions that require unreferenced code. * Remove unused ReSharper directive Unused ReSharper directive in the file IndexTriggers.cs was identified and therefore removed. This change makes the code cleaner and easier to read. * Update activity invocation in workflow runtime Updated the DefaultBackgroundActivityInvoker service in the Elsa.Workflows.Runtime module to annotate the ExecuteAsync method with "RequiresUnreferencedCode" attribute. This change is made considering the potential code trimming issue. Additionally, simplified the process of fetching workflow by directly using FindWorkflowAsync method instead of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods. * Refactor code to simplify workflow definition loading The code for finding and materializing workflow definitions has been simplified. Instead of loading the definition and materializing it into a workflow in separate steps, a new method called FindWorkflowAsync has been introduced to perform both actions at once. This reduces redundancy and makes the code more readable. * Refactor WorkflowHostFactory to streamline workflow creation This commit simplifies the workflow creation process in WorkflowHostFactory. It removes redundant code and extraneous methods, specifically the overloaded CreateAsync method which used WorkflowDefinition. Now, it directly finds and uses the Workflow instance, thereby simplifying the code base and improving maintainability. * Refactor workflow retrieval in WorkflowInstance.cs Changed the way workflow instances are retrieved from the WorkflowDefinitionService. Instead of obtaining the workflow definition and then materializing the workflow from it, the workflow is directly retrieved using the FindWorkflowAsync function. This simplifies the code and avoids unnecessary null-checks. * Remove unnecessary whitespace in WorkflowInstance.cs An extraneous whitespace character was identified and removed in the WorkflowInstance.cs file. This change contributes towards maintaining clean and readable code in the Elsa.ProtoActor module. * Remove unnecessary comment in ProtoActorWorkflowRuntime The unnecessary comment ("Load the workflow definition.") in the method TryStartWorkflowAsync of the ProtoActorWorkflowRuntime.cs file was removed. This is part of an ongoing effort to keep the codebase clean and readable. * Update workflow management features and handlers Added explicit notification handlers for DeleteWorkflowInstances and RefreshActivityRegistry in WorkflowManagementFeature.cs. Also, renamed RefreshActivityRegistryHandler.cs to RefreshActivityRegistry.cs for better clarity. * Add multiple log record support to workflow execution log stores The major change of this commit is the addition of methods to add multiple log records in the WorkflowExecutionLogStore, across different storage modules such as EntityFramework, MongoDB, Dapper, Elasticsearch and Memory. This ensures consistency and uniform behavior across different storage types. Furthermore, some reformatting and tidying up of the code were undertaken to maintain readability and clarity. * Remove redundant workflow definition check The workflow definition existence check and related service retrieval were removed from HttpWorkflowsMiddleware.cs. It was determined that this check was unnecessary as the workflow definition's existence is guaranteed at this point in the process, reducing redundancy in the code. * Improve cancellation token usage in workflow execution This commit refines the usage of cancellation tokens during the execution of workflows in Elsa.Server and HttpWorkflowsMiddleware. Previously, a cancellation token pair was created before ExecuteWithinTimeoutAsync was called, which limited duration control solely to that method. Now, cancellation tokens are included within ExecuteWithinTimeoutAsync method. This allows the method to observe any cancellation initiated by outer scopes, enhancing control over the timeout of operations. * Add PersistStateAsync method to WorkflowHost A new PersistStateAsync method has been added to the WorkflowHost, which enables the host to directly persist its own state. The method has been integrated into the DefaultWorkflowRuntime and HttpWorkflowsMiddleware. This update eliminates the need to continuously get instances of IWorkflowInstanceManager to save state, which improves efficiency and code readability. * Refactor DefaultAlterationRunner service This update simplifies the DefaultAlterationRunner service, reducing the number of code lines and removing unnecessary references. The workflow materialization step has been merged with the find workflow step, and unused namespaces have been dropped. * Refine wording in IWorkflowHost interface documentation The documentation for the 'CanStartWorkflowAsync' method in the IWorkflowHost interface has been cleaned up. The superfluous "or not" verbiage has been removed, making it easier to understand the method's function. * Remove Redis from DistributedCachingTransport The Redis option was removed from the DistributedCachingTransport enumeration. This transport isn't currently implemented. * Remove 'useDistributedCaching' constant The 'useDistributedCaching' constant was removed from `Program.cs`, and conditional logic was updated to use `distributedCachingTransport != DistributedCachingTransport.None`. A new option 'None' was added to the `DistributedCachingTransport` enum to facilitate this change. * Update package tags in MassTransit project file The package tags in the Elsa.Caching.Distributed.MassTransit project file was updated to consolidate the tags, changing 'mass-transit' to 'masstransit'. This change better aligns with standard naming conventions and improves searchability. * Refactor distributed caching implementation This commit involves an extensive refactor of the distributed caching implementation. Distributed caching related code and resources were moved into an independent 'Elsa.Caching.Distributed' module. The interface 'IDistributedChangeTokenSignaler' was deleted and its functionality was replaced by 'IChangeTokenSignalInvoker'. * Refactor order of parameters in GetOrCreateAsync method The order of parameters in the GetOrCreateAsync method within the CachingWorkflowDefinitionStore class has been changed. This change ensures that the `key` parameter is now first, followed by the `factory` parameter. This improves code readability and aligns with standard coding practices. * Refactor cache retrieval in Workflow service Refactoring was done to streamline the way objects are retrieved from cache in the Workflow service. Duplicated code was condensed into a new `GetFromCacheAsync` method, which is now called in the existing methods, thus increasing maintainability and reducing the possibility of errors. * Update method descriptions and fix comments formatting Method descriptions in various contracts have been updated to more accurately reflect their function regarding record addition and updating in the persistence store. All double comment markers (/// ///) have also been corrected to the standard (///) across multiple classes. * Remove unused caching methods in ModuleExtensions The commit removes the unused methods, `UseMemoryCache` and `UseDistributedCache` from the `ModuleExtensions.cs` file. The removal is part of a wider cleanup and refactoring effort to streamline the codebase and improve legibility. * Remove redundant PrimaryKeyName in DapperWorkflowExecutionLogStore The "PrimaryKeyName" constant was removed in DapperWorkflowExecutionLogStore. This change simplifies the initialization of the '_store' property, reducing unnecessary redundancy and complexity. The refactored code maintains the same functionality but improves readability and maintainability. * Refactor SaveAsync methods in Elsa.Dapper Store The SaveAsync functions have been updated in the Store.cs file inside the Elsa.Dapper module. They now include cancellation token parameters and specify that they add or update records, providing clearer distinction and flexibility. * Refactor store initialization in Elsa.Dapper modules Removed the redundant usage of primary keys during the store initialization across Elsa.Dapper module. Simplified the SaveAsync methods by removing the parameter for primary key, making the code cleaner and more maintainable. This refactoring does not affect the module's functionality. * Refactor UserStore in Elsa.Dapper module The code was adjusted to improve readability within the Elsa.Dapper module's UserStore. Two lines that were previously combined have now been separated into distinct lines, making the code structure more clear. * Refactor constructor arguments in MongoDb module Simplified several classes in the MongoDb module by injecting dependencies directly through the constructor instead of assigning them to private readonly fields. This improves readability and removes unnecessary code lines. Also added JetBrains.Annotations where applicable. * Fix comment syntax in IWorkflowInstanceStore A syntax error in the comments for the method SaveManyAsync (in IWorkflowInstanceStore interface) has been corrected. This change ensures that the remarks section of the method is properly formatted and correctly displayed in documentation. * Remove ComputeBookmarkHash from IHttpWorkflowsCacheManager The ComputeBookmarkHash method was removed from IHttpWorkflowsCacheManager to declutter the interface. The functionality was moved and adapted in the HttpWorkflowsMiddleware class to maintain the original functionality. * Add logging to HttpWorkflowsMiddleware In this update, the HttpWorkflowsMiddleware class has been modified to include logging. Specifically, warning logs have been added to track workflow-related processes and to notify if mentioned bookmarks or workflow instances are not found. * Update consumer configuration in MassTransitFeature This commit modifies the consumer configuration in the MassTransitFeature. Instead of hardcoding the consumer type to DispatchCancelWorkflowsRequestConsumer, it now uses the dynamic consumer type retrieved from the context, making the feature more adaptable for different scenarios. * Change default MassTransitBroker to Memory The default value for the variable useMassTransitBroker in Elsa.Server.Web's Program.cs file has been modified. It has been changed from RabbitMq to Memory to change the message broker used by MassTransit in the application. * Remove Datadog.Trace package from Directory.Packages.props The Datadog.Trace package with version 2.49.0 has been removed from the Directory.Packages.props file. This change reflects the fact that this package is no longer required in our project.
2024-04-10 09:51:40 +00:00
runtime.DistributedLockProvider = _ =>
{
switch (distributedLockProviderName)
{
case "Postgres":
return new PostgresDistributedSynchronizationProvider(postgresConnectionString, options =>
{
options.KeepaliveCadence(TimeSpan.FromMinutes(5));
options.UseMultiplexing();
});
case "Redis":
{
var connectionMultiplexer = StackExchange.Redis.ConnectionMultiplexer.Connect(redisConnectionString);
var database = connectionMultiplexer.GetDatabase();
return new RedisDistributedSynchronizationProvider(database);
}
case "File":
return new FileDistributedSynchronizationProvider(new DirectoryInfo(Path.Combine(Directory.GetCurrentDirectory(), "App_Data", "locks")));
case "Noop":
default:
return new NoopDistributedSynchronizationProvider();
}
};
})
.UseEnvironments(environments => environments.EnvironmentsOptions = options => configuration.GetSection("Environments").Bind(options))
.UseScheduling(scheduling =>
{
if (useHangfire)
scheduling.UseHangfireScheduler();
if (useQuartz)
scheduling.UseQuartzScheduler();
})
.UseWorkflowsApi(api =>
{
api.AddFastEndpointsAssembly<Program>();
})
.UseRealTimeWorkflows()
.UseCSharp(options =>
{
options.AppendScript("string Greet(string name) => $\"Hello {name}!\";");
options.AppendScript("string SayHelloWorld() => Greet(\"World\");");
})
.UseJavaScript(options =>
{
options.AllowClrAccess = true;
options.ConfigureEngine(engine =>
{
engine.Execute("function greet(name) { return `Hello ${name}!`; }");
engine.Execute("function sayHelloWorld() { return greet('World'); }");
});
})
.UsePython(python =>
{
python.PythonOptions += options =>
{
// Make sure to configure the path to the python DLL. E.g. /opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/bin/python3.11
// alternatively, you can set the PYTHONNET_PYDLL environment variable.
configuration.GetSection("Scripting:Python").Bind(options);
};
})
.UseLiquid(liquid => liquid.FluidOptions = options => options.Encoder = HtmlEncoder.Default)
.UseHttp(http =>
{
http.ConfigureHttpOptions = options => configuration.GetSection("Http").Bind(options);
Fix WorkflowActivity to Use Cached Workflow Definitions for Consistent Behavior (#5223) * Replace IServiceScopeFactory with IServiceProvider in WorkflowRunner Unused dependencies were removed from the workflow runner service. The IServiceScopeFactory was replaced with IServiceProvider to better handle the creation and deletion of service scopes, resulting in cleaner code with less manual scope management. Microsoft.Extensions.DependencyInjection and System.Diagnostics.CodeAnalysis were removed as they were no longer necessary. * Refactor WorkflowDefinitionActivity to use WorkflowDefinitionService The WorkflowDefinitionActivity class has been refactored to make use of the WorkflowDefinitionService instead of the WorkflowDefinitionStore. This fixes #5222 by ensuring the same activity instances are used in the graph model of the workflow execution context. * Add workflow filtering and caching functionality Added methods to `WorkflowDefinitionService` to find workflow definitions and workflows using filter criteria. A key generation method for caching filtered workflows was also added to `WorkflowDefinitionCacheManager`. The implementation includes generating a hash of the filter parameters and using this hash as a cache key, providing efficient caching functionality for filtered searches. * Refactor TriggerIndexer to handle only ITrigger activities The code in TriggerIndexer has been refactored to deal specifically with ITrigger activities, streamlining its behavior. Removed code related to handling non-ITrigger activities and simplified the workflow creation process. The extraction of "startable" nodes now directly filters and casts to ITrigger, reducing complexity and increasing readability. * Update caching service to support filter-based search The CachingWorkflowDefinitionService has been updated to support workflow definition and workflow search based on filter criteria. The update also includes change of class scope from public to internal. Further, it resolves the missing reference by switching from Elsa.Caching.Contracts to Elsa.Caching. * Optimize Elsa project imports and use explicit cache variable names This commit removes superfluous import references, relocates the 'IChangeTokenSignaler' contract into the 'Elsa.Caching' namespace, and replaces ambiguous 'cache' variable names with more explicit 'memoryCache' across several files. Additionally, new package references have been added and access modifiers have been changed to improve encapsulation. Cleanup enhances readability and maintainability of the codebase. * Add .DotSettings file to Elsa.Caching module A new .DotSettings file has been added to the Elsa.Caching module. This file is used for namespace configuration, specifically to skip the "contracts" folder in code inspections. * Update workflow interfaces to support filter queries The update extends `IWorkflowDefinitionCacheManager` and `IWorkflowDefinitionService` interfaces. Functions are added to allow creating filter cache keys and finding workflow definitions and workflows using a new `WorkflowDefinitionFilter`. This enhances querying flexibility by enabling filtered searches. * Add WorkflowDefinitionVersionId to WorkflowTriggerEqualityComparer A new property, WorkflowDefinitionVersionId, has been added to the object being serialized in WorkflowTriggerEqualityComparer. This change allows for a more accurate comparison between workflow triggers, considering not just the workflow definition ID but also its version. * Update service registration types in WorkflowsFeature Changed the registration type for both IHasher and IBookmarkHasher services from Scoped to Singleton in the workflows feature configuration. This alteration aims to improve application performance and manage service lifetimes more efficiently. * Remove Open.Linq.AsyncExtensions dependency The Open.Linq.AsyncExtensions package reference was removed across the project. The usage within the CachingWorkflowDefinitionStore was updated accordingly to maintain functionality. * Move System.Linq.Dynamic.Core package reference The System.Linq.Dynamic.Core package reference was moved from the Directory.Build.props file to the Elsa.Workflows.Management.csproj file. This change reflects the specific dependency of the Elsa.Workflows.Management module on System.Linq.Dynamic.Core, without impacting other modules. * Implement caching for HTTP workflows This update introduces caching mechanisms for HTTP workflows, which significantly improves their performance. The changes involve creating a `CacheManager` and `CachingHttpWorkflowLookupService`, and modifying some existing components to use the new caching mechanism. Additionally, the `HttpWorkflowsCacheManager` was renamed to `HttpWorkflowsCacheInvalidationManager` to better reflect its role. * Refactor cache management across modules This commit refactor the cache management across various modules. The 'ICacheManager' interface now includes methods for triggering and getting change tokens, and the 'HttpWorkflowsCacheInvalidationManager' has been renamed to 'HttpWorkflowsCacheManager'. The caching functionality in 'WorkflowDefinitionService' and other similar services have been updated to use these new methods, improving consistency and maintainability. * Enable caching in Elsa.Server.Web The "useCaching" variable has been set to true to enable caching. Simultaneously, the method name "UseCachingStores" has been refactored to "UseCache". Conditional statements have been added to check the "useCaching" variable before invoking caching. * Rename method UseCaching to UseCache In the Elsa.Server.Web and Elsa.Http project files, the method UseCaching has been renamed to UseCache. This modification is aimed at bridging naming inconsistencies and maintaining naming standards across the application. * Update HttpCacheFeature class description The class summary for HttpCacheFeature has been revised. Originally, it stated that the class was used for installing services related to HTTP services and activities, but it actually focuses more on HTTP caching. * Remove unused Configure method from HttpCacheFeature The Configure method in HttpCacheFeature was found to be redundant as it wasn't doing any significant work or contributing to any functionality. It has therefore been removed to clean up the code and avoid confusion. * Add 'bug/*' to workflow triggers This commit includes 'bug/*' to the list of triggers in our GitHub Actions workflow. Now, any push or pull request under a 'bug/*' branch will trigger the workflow.
2024-04-15 08:06:27 +00:00
if (useCaching)
http.UseCache();
})
.UseEmail(email => email.ConfigureOptions = options => configuration.GetSection("Smtp").Bind(options))
.UseAlterations(alterations =>
{
if (useMongoDb)
{
// TODO: alterations.UseMongoDb();
}
else if (useDapper)
{
// TODO: alterations.UseDapper();
}
else
{
alterations.UseEntityFrameworkCore(ef =>
{
if (useSqlServer)
ef.UseSqlServer(sqlServerConnectionString);
else if (usePostgres)
ef.UsePostgreSql(postgresConnectionString);
else if (useCockroachDb)
ef.UsePostgreSql(cockroachDbConnectionString!);
else
ef.UseSqlite(sqliteConnectionString);
ef.RunMigrations = runEFCoreMigrations;
});
}
if (useMassTransit)
{
alterations.UseMassTransitDispatcher();
}
2023-10-29 10:36:01 +00:00
})
.UseWorkflowContexts();
if (useQuartz)
{
elsa.UseQuartz(quartz => { quartz.UseSqlite(sqliteConnectionString); });
}
if (useMassTransit)
{
elsa.UseMassTransit(massTransit =>
{
Implement dispatch channels (#4949) * Add DispatchWorkflowOptions and update MassTransit configuration Introduced a new class `DispatchWorkflowOptions` to provide workflow dispatch options. Updated MassTransit configuration to include NET6.0 and NET7.0 support, and ensure correct MassTransit version usage per target framework version. * Add support for configurable MassTransit message dispatching Implemented a feature which allows for configurable MassTransit message dispatching. Added support for specifying channels and message brokers. Message dispatching code was massively refactored and relevant endpoints and response models were updated to support new message dispatching features. * Add IEndpointChannelFormatter interface and implementation An interface for formatting channel queue names, 'IEndpointChannelFormatter', has been added, along with its default implementation 'DefaultEndpointChannelFormatter'. The code that uses hardcoded queue name formatting has been modified to use the new formatter instead, making it more configurable and reusable. The implementation of the formatter uses the 'Humanizer' library to kebab-case the channel names. * Add channel dispatch option to workflow activities Introduced `WorkflowDispatcherChannelOptionsProvider` to provide dropdown channel options for workflow dispatch-related activities. Updated `DispatchWorkflow` and `BulkDispatchWorkflows` activities to include a new dropdown input for specifying a dispatch channel. Also, included the selected channel name in the `DispatchWorkflowOptions` during workflow dispatching process. * Update MassTransit configurations and remove unused code The MassTransit setup in Elsa.MassTransit module has been simplified by removing conditional code for different .NET versions. Additionally, unused parameter '__X_Channel' in 'DispatchWorkflowDefinition' was removed. Lastly, the MassTransit broker in Elsa.Server.Web was switched from RabbitMq to AzureServiceBus. * Refactor dispatch workflow classes and methods Simplified the class, method and variable names related to workflow dispatching in the Elsa.Workflows.Runtime module. For example, the 'WorkflowDispatcherChannelDescriptor' class was renamed to 'DispatcherChannel'. This refactoring was performed to make code more readable and maintainable by removing redundant wording in the naming convention. * Update GitHub Workflow to support feature and issue branches The workflow changes add support for feature and issue branches. Now, it extracts the branch name and verifies the commit exists in the given branch rather than just 'main'. The versioning scheme is also modified to include the branch name and not just the run number. * Add 'bug/*' to triggering branches in packages workflow The 'bug/*' pattern was missing from the triggers that initiate the GitHub actions within our packages workflow. This update includes any branch with a 'bug/' prefix to the list, allowing bug-related branches to start jobs in our CI/CD pipeline. * Remove 'issue/*' and 'bug/*' branches from packages workflow The 'issue/*' and 'bug/*' branches have been removed from the GitHub action workflow for packages. This change was made to simplify the workflow and optimize the triggering of package building. * Update GitHub workflow to handle main branch versioning This commit modifies the GitHub workflow script to accommodate changes when the branch name is "main." If the branch name is "main", a preview version is used. It also updates script execution to print the branch name for easier debugging and verifies commit existence on the correct branch instead of dispatch channels. * Enclose branch names in quotes in packages.yml The update modifies the branch names in the packages.yml GitHub Actions workflow file. The change consists of enclosing the branch names 'main' and 'feature/*' in single quotes, ensuring compatibility and preventing potential string interpretation issues. * Update GitHub workflows package configuration The workflows package configuration has been updated to specifically watch for changes on 'feature/dispatch-channels' rather than on all feature branches. This change will prevent unnecessary builds on less relevant feature branches. * Update trigger branches in packages workflow The triggering branches in the packages workflow have been updated. Previously, only changes in the 'main' and 'feature/dispatch-channels' would trigger the workflow, now any 'feature/*' branch will. This will cause more frequent and comprehensive testing. * Add 'patch/*' to workflow trigger branches This update adds 'patch/*' to the list of branches in .github/workflows/packages.yml that can trigger the workflow. It will allow the workflow to be initiated not just for main and feature branches, but also for patches. * Add 'preview/*' to workflow triggers This commit adds a new trigger for the GitHub Actions workflow. It now also responds to push events on 'preview/*' branches, allowing for automated testing and building of these preview branches. * Update branch name extraction in GitHub Actions The extraction of the branch name has been slightly modified in the packages.yml GitHub workflow file. This alteration ensures the correct branch name is obtained for further processing within the workflow without any discrepancy. * Update branch name extraction in packages.yml Corrected the syntax for extracting the branch name within the packages.yml github workflow file. Added an extra line to print out the ref which might be useful for debugging. * Update branch name extraction in GitHub workflow The commit simplifies the way the branch name is being extracted from the GitHub ref in the packages.yml workflow file. The new method employs straightforward string manipulation, making it easier to understand and debug in case of potential issues. * Add extraction of branch name in workflow Added a new line in the GitHub workflow file (.github/workflows/packages.yml) to extract the last part after the final slash from the branch name. This enhancement allows cleaner naming conventions, especially in cases where branches are named feature/issue-123, as it will only retain 'issue-123'. * Update package naming in Github workflow The Github workflow has been updated to handle package naming more effectively. Previously, the branch name was used directly for package versioning. Now, the last part of the branch name is extracted and used as the package prefix. If the branch name is "main", the package prefix is set to "preview". * Move and add environment variable assignments The placement of the assignment for BRANCH_NAME environment variable was moved for better readability. Additionally, the PACKAGE_PREFIX environment variable was also added. These environment variables are crucial for subsequent steps in the GitHub workflow. * Add workflow dispatch validation and response handling Removed several specific dispatch response classes and consolidated all types of dispatch responses into a single DispatchWorkflowResponse class. Added a new ValidatingWorkflowDispatcher service to validate dispatch requests before they're sent. Updated several classes to work with these changes, including the BackgroundWorkflowDispatcher, MassTransitWorkflowDispatcher, and the API endpoint class. * Handle dispatch workflow failures with exceptions The DispatchWorkflow and BulkDispatchWorkflows activities now throw a FaultException when the dispatch operations fail. Previously, these operations were not checking for success and could fail silently. Now, an unsuccessful dispatch response results in a FaultException with an error message from the dispatch response.
2024-02-16 09:56:30 +00:00
if (useMassTransitBroker == MassTransitBroker.AzureServiceBus)
{
massTransit.UseAzureServiceBus(azureServiceBusConnectionString, serviceBusFeature => serviceBusFeature.ConfigureServiceBus = bus =>
{
bus.PrefetchCount = 100;
bus.LockDuration = TimeSpan.FromMinutes(5);
bus.MaxConcurrentCalls = 32;
bus.MaxDeliveryCount = 8;
// etc.
});
}
Implement dispatch channels (#4949) * Add DispatchWorkflowOptions and update MassTransit configuration Introduced a new class `DispatchWorkflowOptions` to provide workflow dispatch options. Updated MassTransit configuration to include NET6.0 and NET7.0 support, and ensure correct MassTransit version usage per target framework version. * Add support for configurable MassTransit message dispatching Implemented a feature which allows for configurable MassTransit message dispatching. Added support for specifying channels and message brokers. Message dispatching code was massively refactored and relevant endpoints and response models were updated to support new message dispatching features. * Add IEndpointChannelFormatter interface and implementation An interface for formatting channel queue names, 'IEndpointChannelFormatter', has been added, along with its default implementation 'DefaultEndpointChannelFormatter'. The code that uses hardcoded queue name formatting has been modified to use the new formatter instead, making it more configurable and reusable. The implementation of the formatter uses the 'Humanizer' library to kebab-case the channel names. * Add channel dispatch option to workflow activities Introduced `WorkflowDispatcherChannelOptionsProvider` to provide dropdown channel options for workflow dispatch-related activities. Updated `DispatchWorkflow` and `BulkDispatchWorkflows` activities to include a new dropdown input for specifying a dispatch channel. Also, included the selected channel name in the `DispatchWorkflowOptions` during workflow dispatching process. * Update MassTransit configurations and remove unused code The MassTransit setup in Elsa.MassTransit module has been simplified by removing conditional code for different .NET versions. Additionally, unused parameter '__X_Channel' in 'DispatchWorkflowDefinition' was removed. Lastly, the MassTransit broker in Elsa.Server.Web was switched from RabbitMq to AzureServiceBus. * Refactor dispatch workflow classes and methods Simplified the class, method and variable names related to workflow dispatching in the Elsa.Workflows.Runtime module. For example, the 'WorkflowDispatcherChannelDescriptor' class was renamed to 'DispatcherChannel'. This refactoring was performed to make code more readable and maintainable by removing redundant wording in the naming convention. * Update GitHub Workflow to support feature and issue branches The workflow changes add support for feature and issue branches. Now, it extracts the branch name and verifies the commit exists in the given branch rather than just 'main'. The versioning scheme is also modified to include the branch name and not just the run number. * Add 'bug/*' to triggering branches in packages workflow The 'bug/*' pattern was missing from the triggers that initiate the GitHub actions within our packages workflow. This update includes any branch with a 'bug/' prefix to the list, allowing bug-related branches to start jobs in our CI/CD pipeline. * Remove 'issue/*' and 'bug/*' branches from packages workflow The 'issue/*' and 'bug/*' branches have been removed from the GitHub action workflow for packages. This change was made to simplify the workflow and optimize the triggering of package building. * Update GitHub workflow to handle main branch versioning This commit modifies the GitHub workflow script to accommodate changes when the branch name is "main." If the branch name is "main", a preview version is used. It also updates script execution to print the branch name for easier debugging and verifies commit existence on the correct branch instead of dispatch channels. * Enclose branch names in quotes in packages.yml The update modifies the branch names in the packages.yml GitHub Actions workflow file. The change consists of enclosing the branch names 'main' and 'feature/*' in single quotes, ensuring compatibility and preventing potential string interpretation issues. * Update GitHub workflows package configuration The workflows package configuration has been updated to specifically watch for changes on 'feature/dispatch-channels' rather than on all feature branches. This change will prevent unnecessary builds on less relevant feature branches. * Update trigger branches in packages workflow The triggering branches in the packages workflow have been updated. Previously, only changes in the 'main' and 'feature/dispatch-channels' would trigger the workflow, now any 'feature/*' branch will. This will cause more frequent and comprehensive testing. * Add 'patch/*' to workflow trigger branches This update adds 'patch/*' to the list of branches in .github/workflows/packages.yml that can trigger the workflow. It will allow the workflow to be initiated not just for main and feature branches, but also for patches. * Add 'preview/*' to workflow triggers This commit adds a new trigger for the GitHub Actions workflow. It now also responds to push events on 'preview/*' branches, allowing for automated testing and building of these preview branches. * Update branch name extraction in GitHub Actions The extraction of the branch name has been slightly modified in the packages.yml GitHub workflow file. This alteration ensures the correct branch name is obtained for further processing within the workflow without any discrepancy. * Update branch name extraction in packages.yml Corrected the syntax for extracting the branch name within the packages.yml github workflow file. Added an extra line to print out the ref which might be useful for debugging. * Update branch name extraction in GitHub workflow The commit simplifies the way the branch name is being extracted from the GitHub ref in the packages.yml workflow file. The new method employs straightforward string manipulation, making it easier to understand and debug in case of potential issues. * Add extraction of branch name in workflow Added a new line in the GitHub workflow file (.github/workflows/packages.yml) to extract the last part after the final slash from the branch name. This enhancement allows cleaner naming conventions, especially in cases where branches are named feature/issue-123, as it will only retain 'issue-123'. * Update package naming in Github workflow The Github workflow has been updated to handle package naming more effectively. Previously, the branch name was used directly for package versioning. Now, the last part of the branch name is extracted and used as the package prefix. If the branch name is "main", the package prefix is set to "preview". * Move and add environment variable assignments The placement of the assignment for BRANCH_NAME environment variable was moved for better readability. Additionally, the PACKAGE_PREFIX environment variable was also added. These environment variables are crucial for subsequent steps in the GitHub workflow. * Add workflow dispatch validation and response handling Removed several specific dispatch response classes and consolidated all types of dispatch responses into a single DispatchWorkflowResponse class. Added a new ValidatingWorkflowDispatcher service to validate dispatch requests before they're sent. Updated several classes to work with these changes, including the BackgroundWorkflowDispatcher, MassTransitWorkflowDispatcher, and the API endpoint class. * Handle dispatch workflow failures with exceptions The DispatchWorkflow and BulkDispatchWorkflows activities now throw a FaultException when the dispatch operations fail. Previously, these operations were not checking for success and could fail silently. Now, an unsuccessful dispatch response results in a FaultException with an error message from the dispatch response.
2024-02-16 09:56:30 +00:00
if (useMassTransitBroker == MassTransitBroker.RabbitMq)
{
massTransit.UseRabbitMq(rabbitMqConnectionString, rabbit => rabbit.ConfigureServiceBus = bus =>
{
bus.PrefetchCount = 4;
bus.Durable = true;
bus.AutoDelete = false;
bus.ConcurrentMessageLimit = 32;
// etc.
});
}
});
}
Add caching to workflow runtime and workflow management stores (#5174) * Add caching to workflow runtime and workflow management stores The update introduces caching to workflow runtime and workflow management stores to enhance performance. This is achieved by adding decorators for several stores, which cache records to reduce database fetches. Additionally, a signaler for change tokens allows for cache invalidation when changes occur. The MemoryCache feature has also been updated to include Scrutor for decoration and the caching duration can be configured through the new CachingOptions class. * Refactor WorkflowsMiddleware for HTTP Endpoint bookmarks and triggers The WorkflowsMiddleware has been extensively refactored to handle HTTP Endpoint bookmarks and triggers. This involves breaking down the InvokeAsync method by extracting parts of its functionality into separate helper methods such as FindTriggersAsync and FindBookmarksAsync. Moreover, Assist with authorization checks, workflow execution within request timeout, and handling of workflow faults has been improved to be more efficient and clearly segmented. * Update HTTP endpoint authorization to use Workflow context The authorization process in the AuthenticationBasedHttpEndpointAuthorizationHandler class has been updated to use the Workflow context instead of the WorkflowInstanceId string. The AuthorizeHttpEndpointContext model has been correspondingly changed to include a Workflow property, thereby strengthening the link between authorization and specific workflows. * Add FindAsync methods to trigger and bookmark stores The code adjustments add new FindAsync methods to the trigger and bookmark store contracts as well as all their concrete implementations (MongoDb, Memory and EFCore). These methods support fetching the first record matching a given filter. The adjustments also include minor syntax improvements and the addition of [UsedImplicitly] attributes where needed. * Refactor WorkflowsMiddleware for improved workflow handling The code was refactored to simplify the flow of handling workflows in the WorkflowsMiddleware class. Specifically, the methods to start and resume a workflow have been extracted to improve code readability. Further, handleErrorMiddlewares was also updated to better manage instances where no valid workflows or base paths are found. * Add caching functionality to WorkflowsMiddleware Added IMemoryCache usage in the WorkflowsMiddleware to cache lookup results for workflows and their associated triggers. This will reduce the number of database operations required when searching for workflows, thus improving performance. The cache is maintained for one minute before it is refreshed. * Implement dynamic cache duration for workflows The code has been updated to have a dynamic cache duration for the workflows instead of a hardcoded one minute. By using the CachingOptions service, the cache duration can now be set in the configuration making it more flexible and adaptable to different performance needs. * Add HttpWorkflowsCacheManager for caching HTTP workflows This commit includes the implementation of IHttpWorkflowsCacheManager for caching of HTTP workflows. New handlers have been added to invalidate cache on workflow updates. Additionally, WorkflowsMiddleware has been renamed to HttpWorkflowsMiddleware. * Refactor workflow trigger handling and caching The refactoring includes deletion of `IndexWorkflowTriggersHandler.cs` and creation of `IndexTriggers.cs` thus revising the workflow trigger indexing approach. Also, revamped the `ITriggerIndexer` interface which now handles deletion of triggers with specific workflow and filter. Furthermore, the caching mechanism in `HttpWorkflowsCacheManager.cs` is modified to handle eviction of workflow definitions and triggers separately boosting its efficiency. * Add summary to IndexedWorkflowTriggers A summary has been added to the 'IndexedWorkflowTriggers' class, providing a brief description. This description outlines that it represents a collection of indexed workflow triggers, promoting clearer understanding for future reference. * Refactor memory caching feature into separate module This commit separates the memory caching feature from the Elsa.Common module into a distinct Elsa.Caching module. This includes moving and renaming related files, such as the MemoryCacheFeature class and associated dependencies. The references in other modules and in the main solution file have been updated accordingly to include the new Elsa.Caching module. * Add distributed caching and update async methods Introduced a distributed caching feature with extensible change token signal publishing. Updated various cache-related methods to be asynchronous for improved performance and responsiveness. Also updated some workflow identity references for clarity. * Add distributed caching with MassTransit support This addition includes the implementation of a distributed caching system with MassTransit transport. The changes introduce necessary interfaces and services, new distributed caching feature along with the support for MassTransit as a transport option. Moreover, the instance management feature has been renamed to clustering feature for better clarity. * Refactor queue naming and scope of MassTransitChangeTokenSignalPublisher Queue name construction is adjusted in the RabbitMqServiceBusFeature and AzureServiceBusFeature modules for better organization and readability. MassTransitChangeTokenSignalPublisher is now a singleton service, ensuring the signal publisher can be shared across the application, increasing efficiency and performance. Also, introduced use of DistributedCacheFeature in MassTransitDistributedCacheFeature module for better modularization. * Add caching capabilities to workflow definition service This commit introduces caching to the workflow definition service, improving the performance for retrieving workflow definitions. 4 new classes have been created (`CachingWorkflowDefinitionService`, `EvictWorkflowDefinitionServiceCache`, `WorkflowDefinitionCacheManager`, and `IWorkflowDefinitionCacheManager`), and several existing classes have been updated to support caching. The caching also includes invalidation mechanisms, ensuring data consistency. * Refactor caching mechanism in workflow definition In the workflow definition module, the explicit caching functionality related to workflow definition versioning has been removed in favor of a more streamlined approach. Additionally, the manner in which services are registered has been altered. As a result, the caching now directly involves the overall workflow definition rather than individual versions, simplifying the caching logic and potentially improving the performance. * Update MassTransitBroker and enable RealTimeWorkflows and SignalRHubs The MassTransitBroker has been updated to Memory from RabbitMq. In addition, the RealTimeWorkflows and UseWorkflowsSignalRHubs features are now enabled in the code. This change will impact how the service communicates and processes real-time requests for workflow operations. * Reformat variable types in HttpFeature The reformatting involves a list of variable types in the HttpFeature module. Each type now appears on a new line for improved readability, making the code easier to maintain and review. * Update HTTP workflows cache invalidation handler XML comment * Remove unnecessary using directives Unnecessary using directives were deleted across several files in the Elsa.Http module. This simplifies the code and will possibly improve execution speed. Specific deletions include those for Encoding, Unicode, Extensions, Collections.Generic, Linq, Text, and Tasks namespaces. * Refactor HttpWorkflowsMiddleware constructor This commit simplifies the HttpWorkflowsMiddleware class constructor. It removes the intermediary variables `_next` and `_options` and directly uses the passed arguments in the constructor. Now, the `next` and `options` parameters are used directly throughout the middleware. * Simplify workflow retrieval in HttpWorkflowsMiddleware This refactoring replaces the use of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync with a single method, FindWorkflowAsync. This simplifies the middleware code and likely improves performance by reducing the number of database queries or service calls required to retrieve a workflow. * Refactor workflow retrieval in HttpBookmarkProcessor Simplified the workflow retrieval process in HttpBookmarkProcessor.cs. Replaced FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods with a single FindWorkflowAsync call. This reduces the complexity and improves efficiency in retrieving workflow. * Optimize FindWorkflowAsync method in HttpWorkflowsCacheManager Removed redundant lines of code to simplify workflow search functionality. This change simplified the FindWorkflowAsync method by directly calling the FindWorkflowAsync function in the workflowDefinitionService, thus increasing code readability and efficiency. * Refactor Endpoint.cs for workflow retrieval The method for obtaining a workflow in the Endpoint.cs script has been refactored and streamlined. The 'GetWorkflowDefinition' method is replaced by the 'GetWorkflowAsync' method which directly retrieves the workflow, without the intermediate step of materializing the workflow definition. This shortens the code and simplifies the process. * Refactor InputFunctionsDefinitionProvider constructor The constructor for InputFunctionsDefinitionProvider has been simplified by removing unnecessary private fields. Services are now directly used in the method instead of being stored in fields. This improves readability and reduces complexity in the class structure. * Refactor WorkflowInstance with improved state handling Simplified the methods for handling workflow and workflow state in the WorkflowInstance class. The refactoring also included some code clean-ups and variable renaming. The new implementation provides better readability and maintainability of the code by reducing unnecessary lines and improving structuring of objects and responses. * Remove unused IBookmarkManager and update workflow functions IBookmarkManager from ProtoActorWorkflowRuntime.cs file is removed due to its redundant status. Additionally, the "FindAsync" method has been updated to use a cancellation token. Also, annotations were added to the "ExportWorkflowStateAsync" and "ImportWorkflowStateAsync" methods to flag calls to functions that require unreferenced code. * Remove unused ReSharper directive Unused ReSharper directive in the file IndexTriggers.cs was identified and therefore removed. This change makes the code cleaner and easier to read. * Update activity invocation in workflow runtime Updated the DefaultBackgroundActivityInvoker service in the Elsa.Workflows.Runtime module to annotate the ExecuteAsync method with "RequiresUnreferencedCode" attribute. This change is made considering the potential code trimming issue. Additionally, simplified the process of fetching workflow by directly using FindWorkflowAsync method instead of FindWorkflowDefinitionAsync and MaterializeWorkflowAsync methods. * Refactor code to simplify workflow definition loading The code for finding and materializing workflow definitions has been simplified. Instead of loading the definition and materializing it into a workflow in separate steps, a new method called FindWorkflowAsync has been introduced to perform both actions at once. This reduces redundancy and makes the code more readable. * Refactor WorkflowHostFactory to streamline workflow creation This commit simplifies the workflow creation process in WorkflowHostFactory. It removes redundant code and extraneous methods, specifically the overloaded CreateAsync method which used WorkflowDefinition. Now, it directly finds and uses the Workflow instance, thereby simplifying the code base and improving maintainability. * Refactor workflow retrieval in WorkflowInstance.cs Changed the way workflow instances are retrieved from the WorkflowDefinitionService. Instead of obtaining the workflow definition and then materializing the workflow from it, the workflow is directly retrieved using the FindWorkflowAsync function. This simplifies the code and avoids unnecessary null-checks. * Remove unnecessary whitespace in WorkflowInstance.cs An extraneous whitespace character was identified and removed in the WorkflowInstance.cs file. This change contributes towards maintaining clean and readable code in the Elsa.ProtoActor module. * Remove unnecessary comment in ProtoActorWorkflowRuntime The unnecessary comment ("Load the workflow definition.") in the method TryStartWorkflowAsync of the ProtoActorWorkflowRuntime.cs file was removed. This is part of an ongoing effort to keep the codebase clean and readable. * Update workflow management features and handlers Added explicit notification handlers for DeleteWorkflowInstances and RefreshActivityRegistry in WorkflowManagementFeature.cs. Also, renamed RefreshActivityRegistryHandler.cs to RefreshActivityRegistry.cs for better clarity. * Add multiple log record support to workflow execution log stores The major change of this commit is the addition of methods to add multiple log records in the WorkflowExecutionLogStore, across different storage modules such as EntityFramework, MongoDB, Dapper, Elasticsearch and Memory. This ensures consistency and uniform behavior across different storage types. Furthermore, some reformatting and tidying up of the code were undertaken to maintain readability and clarity. * Remove redundant workflow definition check The workflow definition existence check and related service retrieval were removed from HttpWorkflowsMiddleware.cs. It was determined that this check was unnecessary as the workflow definition's existence is guaranteed at this point in the process, reducing redundancy in the code. * Improve cancellation token usage in workflow execution This commit refines the usage of cancellation tokens during the execution of workflows in Elsa.Server and HttpWorkflowsMiddleware. Previously, a cancellation token pair was created before ExecuteWithinTimeoutAsync was called, which limited duration control solely to that method. Now, cancellation tokens are included within ExecuteWithinTimeoutAsync method. This allows the method to observe any cancellation initiated by outer scopes, enhancing control over the timeout of operations. * Add PersistStateAsync method to WorkflowHost A new PersistStateAsync method has been added to the WorkflowHost, which enables the host to directly persist its own state. The method has been integrated into the DefaultWorkflowRuntime and HttpWorkflowsMiddleware. This update eliminates the need to continuously get instances of IWorkflowInstanceManager to save state, which improves efficiency and code readability. * Refactor DefaultAlterationRunner service This update simplifies the DefaultAlterationRunner service, reducing the number of code lines and removing unnecessary references. The workflow materialization step has been merged with the find workflow step, and unused namespaces have been dropped. * Refine wording in IWorkflowHost interface documentation The documentation for the 'CanStartWorkflowAsync' method in the IWorkflowHost interface has been cleaned up. The superfluous "or not" verbiage has been removed, making it easier to understand the method's function. * Remove Redis from DistributedCachingTransport The Redis option was removed from the DistributedCachingTransport enumeration. This transport isn't currently implemented. * Remove 'useDistributedCaching' constant The 'useDistributedCaching' constant was removed from `Program.cs`, and conditional logic was updated to use `distributedCachingTransport != DistributedCachingTransport.None`. A new option 'None' was added to the `DistributedCachingTransport` enum to facilitate this change. * Update package tags in MassTransit project file The package tags in the Elsa.Caching.Distributed.MassTransit project file was updated to consolidate the tags, changing 'mass-transit' to 'masstransit'. This change better aligns with standard naming conventions and improves searchability. * Refactor distributed caching implementation This commit involves an extensive refactor of the distributed caching implementation. Distributed caching related code and resources were moved into an independent 'Elsa.Caching.Distributed' module. The interface 'IDistributedChangeTokenSignaler' was deleted and its functionality was replaced by 'IChangeTokenSignalInvoker'. * Refactor order of parameters in GetOrCreateAsync method The order of parameters in the GetOrCreateAsync method within the CachingWorkflowDefinitionStore class has been changed. This change ensures that the `key` parameter is now first, followed by the `factory` parameter. This improves code readability and aligns with standard coding practices. * Refactor cache retrieval in Workflow service Refactoring was done to streamline the way objects are retrieved from cache in the Workflow service. Duplicated code was condensed into a new `GetFromCacheAsync` method, which is now called in the existing methods, thus increasing maintainability and reducing the possibility of errors. * Update method descriptions and fix comments formatting Method descriptions in various contracts have been updated to more accurately reflect their function regarding record addition and updating in the persistence store. All double comment markers (/// ///) have also been corrected to the standard (///) across multiple classes. * Remove unused caching methods in ModuleExtensions The commit removes the unused methods, `UseMemoryCache` and `UseDistributedCache` from the `ModuleExtensions.cs` file. The removal is part of a wider cleanup and refactoring effort to streamline the codebase and improve legibility. * Remove redundant PrimaryKeyName in DapperWorkflowExecutionLogStore The "PrimaryKeyName" constant was removed in DapperWorkflowExecutionLogStore. This change simplifies the initialization of the '_store' property, reducing unnecessary redundancy and complexity. The refactored code maintains the same functionality but improves readability and maintainability. * Refactor SaveAsync methods in Elsa.Dapper Store The SaveAsync functions have been updated in the Store.cs file inside the Elsa.Dapper module. They now include cancellation token parameters and specify that they add or update records, providing clearer distinction and flexibility. * Refactor store initialization in Elsa.Dapper modules Removed the redundant usage of primary keys during the store initialization across Elsa.Dapper module. Simplified the SaveAsync methods by removing the parameter for primary key, making the code cleaner and more maintainable. This refactoring does not affect the module's functionality. * Refactor UserStore in Elsa.Dapper module The code was adjusted to improve readability within the Elsa.Dapper module's UserStore. Two lines that were previously combined have now been separated into distinct lines, making the code structure more clear. * Refactor constructor arguments in MongoDb module Simplified several classes in the MongoDb module by injecting dependencies directly through the constructor instead of assigning them to private readonly fields. This improves readability and removes unnecessary code lines. Also added JetBrains.Annotations where applicable. * Fix comment syntax in IWorkflowInstanceStore A syntax error in the comments for the method SaveManyAsync (in IWorkflowInstanceStore interface) has been corrected. This change ensures that the remarks section of the method is properly formatted and correctly displayed in documentation. * Remove ComputeBookmarkHash from IHttpWorkflowsCacheManager The ComputeBookmarkHash method was removed from IHttpWorkflowsCacheManager to declutter the interface. The functionality was moved and adapted in the HttpWorkflowsMiddleware class to maintain the original functionality. * Add logging to HttpWorkflowsMiddleware In this update, the HttpWorkflowsMiddleware class has been modified to include logging. Specifically, warning logs have been added to track workflow-related processes and to notify if mentioned bookmarks or workflow instances are not found. * Update consumer configuration in MassTransitFeature This commit modifies the consumer configuration in the MassTransitFeature. Instead of hardcoding the consumer type to DispatchCancelWorkflowsRequestConsumer, it now uses the dynamic consumer type retrieved from the context, making the feature more adaptable for different scenarios. * Change default MassTransitBroker to Memory The default value for the variable useMassTransitBroker in Elsa.Server.Web's Program.cs file has been modified. It has been changed from RabbitMq to Memory to change the message broker used by MassTransit in the application. * Remove Datadog.Trace package from Directory.Packages.props The Datadog.Trace package with version 2.49.0 has been removed from the Directory.Packages.props file. This change reflects the fact that this package is no longer required in our project.
2024-04-10 09:51:40 +00:00
if (distributedCachingTransport != DistributedCachingTransport.None)
{
elsa.UseDistributedCache(distributedCaching =>
{
if (distributedCachingTransport == DistributedCachingTransport.MassTransit) distributedCaching.UseMassTransit();
});
}
elsa.InstallDropIns(options => options.DropInRootDirectory = Path.Combine(Directory.GetCurrentDirectory(), "App_Data", "DropIns"));
elsa.AddSwagger();
elsa.AddFastEndpointsAssembly<Program>();
Add Component Testing Framework (#5261) * Add application component tests Multiple new test files were added to deliver application component tests. This move improves testing by adding integration tests that cover overall system behavior and checking end-to-end actions. Ensuring the system functions correctly as a whole. In the process, updating some package versions to maintain compatibility. * Add RefitSettings helper and revise API client service configuration The commit introduces a 'RefitSettingsHelper' for Elsa API client and revises the way the API client services are configured. It also makes improvements to the WorkflowServerTestWebAppFactory for component testing. Some endpoint contracts related to workflow execution are also updated to have optional parameters. * Remove old tests and add new workflow tests This commit removes old, unnecessary tests and incorporates new workflow tests. It also improves the Elsa API client JSON serializer and adds a helper for HttpResponseMessage. Lastly, the commit introduces changes to properly configure the test logging and to manage application settings. * Add HttpHelloWorld workflow tests A new component test scenario, HttpHelloWorldTests, has been created for testing an HttpHelloWorld workflow. This involves asserting if a workflow responds correctly with "Hello World". Furthermore, an HTTP workflow client has been introduced in the WorkflowServerTestWebAppFactory class to provide a base address for workflow API calls. * Add new test file and update workflow execution tests This change adds a new test file "fork-1.json" to the Elsa.Workflows.Api.ComponentTests project. Also, updates were made throughout the tests to replace the WorkflowServerTestWebAppFactory with a fixture, allowing the tests to run in parallel. Lastly, unnecessary warning suppression was removed from the Elsa.Workflows.Core extension method. * Add filter for .json and .elsa files in BlobStorageWorkflowProvider This change adds a BrowseFilter in the BlobStorageWorkflowProvider options. This filter checks for files that end with .json or .elsa and includes only these files when browsing through the blob storage. This filter helps prioritize specific workflow file types. * Rename WorkflowServerTestWebAppFactoryFixture and update usage The old class name "WorkflowServerTestWebAppFactoryFixture" has been replaced with the more accurate "WorkflowServerWebAppFactoryFixture". All references to the previous name in other classes were also updated accordingly. In addition, the directory key in the method "CreateConvoyOptionsBuilder" has been updated from "Workflows" to "Scenarios". * Update test fixture in workflow tests The commit updates the test fixture in two test classes: HttpHelloWorldTests and HelloWorldTests. The former test fixture, WorkflowServerTestWebAppFactoryFixture, was replaced by WorkflowServerWebAppFactoryFixture to accurately match the testing needs. * Update .csproj file paths and reorganize tests The commit modifies the file paths for several test scenario files in the Elsa.Workflows.Api.ComponentTests.csproj, reflecting a reorganization of the tests. Previously static paths have been updated to new paths under 'Scenarios'. Additionally, two new test files related to 'LogPersistenceModes' have been included in the project. * Add tests for log persistence modes This commit introduces two new test scenarios for logging persistence modes and includes a related test called 'HelloWorldWorkflow'. These tests cover scenarios where certain workflow inputs should be stored and others shouldn't, thereby testing the log persistence feature. This ensures that the logging behavior respects the specified persistence mode. * Add log persistence tests and update LogPersistenceMode enum The commit contains the addition of new log persistence tests for verifying correctness of log persistence behavior. Furthermore, the LogPersistenceMode enum has been updated, replacing 'Default' with 'Inherit'. This change makes the mode's purpose clearer. Lastly, new test scenarios and test data files were added for more comprehensive testing. * Remove obsolete component tests and support files The files removed are no longer necessary for the current state of the application. They include various component tests and their related support files within the Elsa.Workflows.Api.ComponentTests project. By removing these, the project structure is cleaner and only contains relevant tests. * Add dispatch workflow scenario tests and necessary helper classes This commit includes two new tests for dispatching workflows, along with the creation of new 'ChildWorkflow' and 'DispatchAndWaitWorkflow' classes. Auxiliary helpers and services have been added to aid in managing workflow events and signals for these tests. The 'ComponentTest' has also been upgraded to support disposal handling. * Remove ITestOutputHelper dependency from test classes Removed the dependency on ITestOutputHelper in multiple test classes across various workflow scenarios. This change simplifies the test class constructors by reducing the number of required dependencies, contributing to cleaner and leaner code. * Add 'Hello World' scenario to WorkflowCompletion tests The 'Hello World' scenario was moved into WorkflowCompletion tests, along with changes in workflow definition identifiers. As part of these changes, the 'hello-world.json' file was updated; a new file under the same name was created in the WorkflowCompletion area and the workflow identifiers in basic and workflow completion tests were updated accordingly. Additionally, 'fork-1.json' has been renamed to 'fork.json'. * Add support for cluster hosting tests This commit introduces a suite of integration tests designed to validate the behaviour of hosting multiple instances of Elsa in a clustered environment. These tests simulate a typical clustered hosting scenario by using 'App', 'Cluster', and 'Infrastructure' objects to emulate different instances of the Elsa workflow engine running on separate servers. Name changes were made to certain classes and methods to reflect their new scopes and roles within the testing environment. * Add performance tests and improve component tests Added a new performance tests project scaffold, complete with its own project file, build properties file, and a dummy test. Updated component tests to improve multi-pod testing, primarily through the addition of additional service scopes and asserting activity registry synchronization. These changes also required updates to existing project and props files as well as the solution file. * Update ActivityRegistrySyncTests and Infrastructure Added a reference to Services in ActivityRegistrySyncTests and removed unnecessary whitespace in both files. The test component Elsa.Workflows has been modified to import newly added services, ensuring all tests are running with the expected resources and services. * Fix comment * Add NOOP implementations for stores * Update PostgreSQL image and adjust test timings The PostgreSQL image used for testing has been updated to the latest version from 13.3-alpine. Timeouts in ISignalManager and DispatchWorkflowsTests have been reduced for efficiency. A delay in the ChildWorkflow has also been decreased. Additionally, an 'ImportWorkflowActivity' test in ActivityRegistrySyncTests has been marked as not yet implemented.
2024-04-26 13:49:14 +00:00
ConfigureForTest?.Invoke(elsa);
});
Fix Race Condition in WorkflowDefinitionActivity (#5315) * Refactor hash method in Hasher.cs The Hash method in the Hasher.cs file has been refactored to use more efficient code. Instead of creating a SHA256 instance, it directly hashes the data using SHA256.HashData. As a result, the private Hash method and its usage, which only has relevance to the previous approach, have been removed. * Replace GetHashCode with SHA256 for script caching Two unused namespaces, Esprima and Esprima.Ast, have been removed, and two new ones, System.Security.Cryptography and System.Text, have been added. These changes in Elsa.JavaScript's JintJavaScriptEvaluator file were made to replace the basic GetHashCode method, which was used earlier to generate a cache key for JavaScript scripts, with SHA256 hash function to ensure uniqueness and avoid possible hash collisions. This greatly increases the reliability of the caching mechanism. * Set HTTP client timeout in WorkflowServer The client timeout in WorkflowServer has been set to 1 minute for `ResrService` and `client` methods. This was done to manage long running requests and prevent timeouts. * Refactor code and add validation in ScheduleActivityAsync methods An unnecessary line of code in the ScheduleActivityAsync method under the ActivityExecutionContext file has been removed to simplify the function. Meanwhile, validation has been added to ensure that specified activities are part of the workflow. This improves code clarity and prevents potential errors caused by incorrect activity scheduling. * Refactor ActivityVisitor to use ActivityVisitorContext The ActivityVisitor class in Elsa.Workflows.Core was refactored to use an ActivityVisitorContext class. The introduction of this context class replaced the multiple hashsets that were used within function signatures, consolidating them into a single object and reducing function complexity. This change improves readability and code organization. * Refactor WorkflowDefinitionActivity to use WorkflowGraph Adjusted the WorkflowDefinitionActivity class in the Elsa.Workflows.Management module. The changes include migration from using Workflow to WorkflowGraph objects and condensing code blocks for clearer readability. Additionally, a new 'IsInitialized' field is added to eliminate potential race conditions during the graph construction process. * Add activity validation to WorkflowExecutionContext A check has been implemented in the WorkflowExecutionContextExtensions to validate that a specified activity is part of the workflow. This prevents incorrect activity references when scheduling tasks in the workflow. * Refactor to use WorkflowGraph instead of Workflow The codebase has been refactored to utilize the WorkflowGraph instead of the Workflow while running and manipulating workflows. Additional changes include restructuring WorkflowRunner and WorkflowDefinitionService classes, introducing WorkflowGraphBuilder usage, and mapping updates in WorkflowDefinitionMapper. The WorkflowHost, WorkflowDispatcher, and WorkflowExecutionContext have also been updated accordingly. * Add WorkflowGraph and related services This commit introduces the IWorkflowGraphBuilder interface, the WorkflowGraph model, and an implementation of the interface in the WorkflowGraphBuilder class. The purpose of these additions is to establish the building and structure of a workflow graph. The WorkflowGraph model also includes activity node handling and hashing capabilities. * Add tests to ensure exception when scheduling an activity not part of workflow Several unit tests were written to ensure that the right behavior is exhibited when scheduling an activity that is not part of the workflow. An exception is expected to be thrown in this case. Additionally, new service definitions, workflow definitions and workflow queries were added for more comprehensive testing. * Update WorkflowGraph class and add comment descriptions This commit updates the WorkflowGraph class by extending its descriptions and implementing an explicit mention to the Workflow reference. Additionally, more attribute descriptions have been added to increase code readability and comprehension. * Removed unnecessary brackets * Remove unused services from workflow management This commit deletes "ExpressionDescriptorRegistryPopulator.cs" and "ScopedWorkflowDefinitionLookup.cs" files from Elsa.Workflows.Management.Services. These files, containing obsolete services, are no longer used in the workflow management process. The services' registration has been removed as well from "WorkflowManagementFeature.cs". * Remove TestWorkflowDefinitionService.cs from component tests A file, TestWorkflowDefinitionService.cs, was removed under Elsa.Workflows.ComponentTests. This is part of the improvement process where inefficient or unnecessary test files are cleaned up. * Add comments to WorkflowDefinitionActivityTests This commit adds explanatory comments to the WorkflowDefinitionActivityTests file. The comments provide information about the purpose of these tests and a reference to a related issue in the project's Github repository. * Updated duplicate and missing package references --------- Co-authored-by: Raymond den Haan <raymond.den.haan@nexxbiz.io>
2024-05-01 12:53:37 +00:00
services.Configure<CachingOptions>(options => options.CacheDuration = TimeSpan.FromDays(1));
services.AddHealthChecks();
2023-08-14 12:34:11 +00:00
services.AddControllers();
2023-07-25 19:02:07 +00:00
services.AddCors(cors => cors.AddDefaultPolicy(policy => policy.AllowAnyHeader().AllowAnyMethod().AllowAnyOrigin().WithExposedHeaders("*")));
// Build the web application.
var app = builder.Build();
2022-07-26 21:30:42 +00:00
// app.UseSimulatedLatency(
// TimeSpan.FromMilliseconds(1000),
// TimeSpan.FromMilliseconds(3000)
// );
// Configure the pipeline.
if (app.Environment.IsDevelopment())
app.UseDeveloperExceptionPage();
// CORS.
app.UseCors();
// Health checks.
app.MapHealthChecks("/");
// Routing used for SignalR.
app.UseRouting();
// Security.
app.UseAuthentication();
app.UseAuthorization();
// Elsa API endpoints for designer.
var routePrefix = app.Services.GetRequiredService<IOptions<HttpActivityOptions>>().Value.ApiRoutePrefix;
app.UseWorkflowsApi(routePrefix);
// Captures unhandled exceptions and returns a JSON response.
app.UseJsonSerializationErrorHandler();
// Elsa HTTP Endpoint activities.
app.UseWorkflows();
// Swagger API documentation.
if (app.Environment.IsDevelopment())
2023-11-27 10:26:16 +00:00
{
app.UseSwaggerUI();
2023-11-27 10:26:16 +00:00
}
// SignalR.
app.UseWorkflowsSignalRHubs();
// Run.
Add Component Testing Framework (#5261) * Add application component tests Multiple new test files were added to deliver application component tests. This move improves testing by adding integration tests that cover overall system behavior and checking end-to-end actions. Ensuring the system functions correctly as a whole. In the process, updating some package versions to maintain compatibility. * Add RefitSettings helper and revise API client service configuration The commit introduces a 'RefitSettingsHelper' for Elsa API client and revises the way the API client services are configured. It also makes improvements to the WorkflowServerTestWebAppFactory for component testing. Some endpoint contracts related to workflow execution are also updated to have optional parameters. * Remove old tests and add new workflow tests This commit removes old, unnecessary tests and incorporates new workflow tests. It also improves the Elsa API client JSON serializer and adds a helper for HttpResponseMessage. Lastly, the commit introduces changes to properly configure the test logging and to manage application settings. * Add HttpHelloWorld workflow tests A new component test scenario, HttpHelloWorldTests, has been created for testing an HttpHelloWorld workflow. This involves asserting if a workflow responds correctly with "Hello World". Furthermore, an HTTP workflow client has been introduced in the WorkflowServerTestWebAppFactory class to provide a base address for workflow API calls. * Add new test file and update workflow execution tests This change adds a new test file "fork-1.json" to the Elsa.Workflows.Api.ComponentTests project. Also, updates were made throughout the tests to replace the WorkflowServerTestWebAppFactory with a fixture, allowing the tests to run in parallel. Lastly, unnecessary warning suppression was removed from the Elsa.Workflows.Core extension method. * Add filter for .json and .elsa files in BlobStorageWorkflowProvider This change adds a BrowseFilter in the BlobStorageWorkflowProvider options. This filter checks for files that end with .json or .elsa and includes only these files when browsing through the blob storage. This filter helps prioritize specific workflow file types. * Rename WorkflowServerTestWebAppFactoryFixture and update usage The old class name "WorkflowServerTestWebAppFactoryFixture" has been replaced with the more accurate "WorkflowServerWebAppFactoryFixture". All references to the previous name in other classes were also updated accordingly. In addition, the directory key in the method "CreateConvoyOptionsBuilder" has been updated from "Workflows" to "Scenarios". * Update test fixture in workflow tests The commit updates the test fixture in two test classes: HttpHelloWorldTests and HelloWorldTests. The former test fixture, WorkflowServerTestWebAppFactoryFixture, was replaced by WorkflowServerWebAppFactoryFixture to accurately match the testing needs. * Update .csproj file paths and reorganize tests The commit modifies the file paths for several test scenario files in the Elsa.Workflows.Api.ComponentTests.csproj, reflecting a reorganization of the tests. Previously static paths have been updated to new paths under 'Scenarios'. Additionally, two new test files related to 'LogPersistenceModes' have been included in the project. * Add tests for log persistence modes This commit introduces two new test scenarios for logging persistence modes and includes a related test called 'HelloWorldWorkflow'. These tests cover scenarios where certain workflow inputs should be stored and others shouldn't, thereby testing the log persistence feature. This ensures that the logging behavior respects the specified persistence mode. * Add log persistence tests and update LogPersistenceMode enum The commit contains the addition of new log persistence tests for verifying correctness of log persistence behavior. Furthermore, the LogPersistenceMode enum has been updated, replacing 'Default' with 'Inherit'. This change makes the mode's purpose clearer. Lastly, new test scenarios and test data files were added for more comprehensive testing. * Remove obsolete component tests and support files The files removed are no longer necessary for the current state of the application. They include various component tests and their related support files within the Elsa.Workflows.Api.ComponentTests project. By removing these, the project structure is cleaner and only contains relevant tests. * Add dispatch workflow scenario tests and necessary helper classes This commit includes two new tests for dispatching workflows, along with the creation of new 'ChildWorkflow' and 'DispatchAndWaitWorkflow' classes. Auxiliary helpers and services have been added to aid in managing workflow events and signals for these tests. The 'ComponentTest' has also been upgraded to support disposal handling. * Remove ITestOutputHelper dependency from test classes Removed the dependency on ITestOutputHelper in multiple test classes across various workflow scenarios. This change simplifies the test class constructors by reducing the number of required dependencies, contributing to cleaner and leaner code. * Add 'Hello World' scenario to WorkflowCompletion tests The 'Hello World' scenario was moved into WorkflowCompletion tests, along with changes in workflow definition identifiers. As part of these changes, the 'hello-world.json' file was updated; a new file under the same name was created in the WorkflowCompletion area and the workflow identifiers in basic and workflow completion tests were updated accordingly. Additionally, 'fork-1.json' has been renamed to 'fork.json'. * Add support for cluster hosting tests This commit introduces a suite of integration tests designed to validate the behaviour of hosting multiple instances of Elsa in a clustered environment. These tests simulate a typical clustered hosting scenario by using 'App', 'Cluster', and 'Infrastructure' objects to emulate different instances of the Elsa workflow engine running on separate servers. Name changes were made to certain classes and methods to reflect their new scopes and roles within the testing environment. * Add performance tests and improve component tests Added a new performance tests project scaffold, complete with its own project file, build properties file, and a dummy test. Updated component tests to improve multi-pod testing, primarily through the addition of additional service scopes and asserting activity registry synchronization. These changes also required updates to existing project and props files as well as the solution file. * Update ActivityRegistrySyncTests and Infrastructure Added a reference to Services in ActivityRegistrySyncTests and removed unnecessary whitespace in both files. The test component Elsa.Workflows has been modified to import newly added services, ensuring all tests are running with the expected resources and services. * Fix comment * Add NOOP implementations for stores * Update PostgreSQL image and adjust test timings The PostgreSQL image used for testing has been updated to the latest version from 13.3-alpine. Timeouts in ISignalManager and DispatchWorkflowsTests have been reduced for efficiency. A delay in the ChildWorkflow has also been decreased. Additionally, an 'ImportWorkflowActivity' test in ActivityRegistrySyncTests has been marked as not yet implemented.
2024-04-26 13:49:14 +00:00
app.Run();
/// <summary>
/// The main entry point for the application made public for end to end testing.
/// </summary>
[UsedImplicitly]
public partial class Program
{
/// <summary>
/// Set by the test runner to configure the module for testing.
/// </summary>
public static Action<IModule>? ConfigureForTest { get; set; }
}