Find a file
Sipke Schoorstra f53e024d25
Add Elsa.Kafka Module for Kafka Integration with Message Sending and Receiving Activities (#6108)
* Add Kafka module with integration and example setup

Introduced the Kafka module providing consumer integration and activities into the project. This includes new classes for consumer handling, configuration, and activities. An example setup using Docker Compose is also added to facilitate development and testing.

* Refactor KafkaTransportMessage to inline Timestamp namespace

Simplify the namespace usage for the Timestamp type within the KafkaTransportMessage record. This change eliminates the need for a separate using directive for Timestamp, enhancing code readability and maintainability.

* Add support for handling Kafka transport messages

This commit introduces the capability to handle and trigger workflows based on Kafka transport messages. It adds a new handler, notifications, and updates the message stimulus to include correlating fields. Additionally, the Kafka consumers are now managed more modularly with updated startup tasks and mediator integration.

* Enable Kafka integration and fix Kafka options naming

Added support for Kafka integration in Elsa.Server.Web by setting up Kafka configurations in appsettings.json and updating Program.cs. Also, renamed `ConsumerConfigs` to `ConsumerDefinitions` in Kafka options for clarity.

* Add consumer definition enumeration and dropdown support

Introduced `IConsumerDefinitionEnumerator` for managing consumer definitions across providers and implemented in `ConsumerDefinitionEnumerator` class. Enhanced `KafkaFeature` to register these services and updated the `MessageReceived` activity to use a dropdown UI hint for consuming definitions. Improved `StartConsumersTask` by refactoring consumer definition retrieval logic.

* Add SendMessage activity and refine Kafka messaging

Introduce a new SendMessage activity for Kafka, enabling message publishing to specific topics. Refine KafkaTransportMessage model by removing headers and timestamp fields. Adjust the StimulusSender logic to streamline the bookmark queuing process and fix key-value pairing in dropdown options. Update appsettings for corrected Kafka bootstrap server and topic configurations.

* Add producer and topic management support

Introduced interfaces and implementations for managing producer and topic definitions along with their respective enumerators and list providers. Updated `SendMessage` activity to include producer selection and refactored consumer definition providers for better consistency.

* Refactor Kafka configuration property names

Renamed Kafka configuration properties for better consistency and readability across the codebase. Updated property names from `ProducerDefinitions` to `Producers`, `ConsumerDefinitions` to `Consumers`, and `TopicDefinitions` to `Topics`. Added missing input attribute in `SendMessage.cs` and registered additional handlers in `KafkaFeature.cs`.

* Add custom serializers for Kafka message handling

Introduced `DefaultSerializers` class for custom serialization and deserialization of Kafka messages. Updated `MessageReceived` and `SendMessage` activities to use these custom serializers, and modified `KafkaOptions` to include them.

* Fix ExpandoObject serialization method parameter

Changed the serialization type from `ExpandoObject` to the actual type of the object to ensure proper serialization. This ensures that derived types are correctly handled during the serialization process.

* Add Producer and Consumer workflows for Kafka

Introduced two new workflows: `ProducerWorkflow` and `ConsumerWorkflow` for handling Kafka messages. Updated `DefaultSerializers` to use camelCase property naming and modified `appsettings.json` to include `topic-2` and format entries.

* Add JSON serialization to log output in ConsumerWorkflow

This change enhances the log output by serializing messages to JSON format before writing them. The addition of System.Text.Json ensures that the message content is presented in a structured and standardized format in logs.

* Add correlation strategies and update Kafka features

Implemented HeaderCorrelationStrategy and NullCorrelationStrategy, and updated KafkaFeature to support customizable correlation strategies. Added correlation ID handling to Kafka transport messages and updated config and handlers accordingly.

* Add tenant accessor to ConsumerDefinitionWorkflowContextProvider

Integrated ITenantAccessor to the provider to support tenant-specific context loading. Updated the constructor and LoadAsync method to retrieve the tenant information and use it for context-specific operations.

* Switch to MySQL and disable Kafka

This commit changes the SQL database provider from SQLite to MySQL and disables Kafka use. It also includes necessary adjustments such as adding MySQL handling in configuration and connection setups, updating `docker-compose` to include MySQL services, and referencing MySQL projects in the `.csproj` file.

* Add UI property handlers to multiple features

This commit introduces various UI property handlers across several features such as Python, JavaScript, CSharp, and Workflow features to enhance user interface property handling. It also updates the property UI handler resolution logic to better manage cases where providers are not available. Furthermore, adjustments were made in the server configuration to switch database providers and enable Kafka.

* Refactor property UI handler retrieval logic

Modified the logic to fetch property UI handlers by preloading them into a list and then filtering. This change improves readability and potentially performance by reducing repetitive service provider calls.

* Incremental work on Kafka workers and predicate evaluation

* Merge BookmarkInvoker with BookmarkResumer

* Register IWorkerManager

* Change lifetime scope of WorkerManager to Singleton

* **Introduce topic subscription handling for Kafka workers**

Added `IWorkerTopicSubscriber` interface and its implementation for managing topic subscriptions. Enhanced workers to bind triggers and bookmarks dynamically based on existing data. Updated several classes and methods to support topic-based subscriptions and headers.

* Refactor trigger matching logic.

Extract trigger matching conditions into `IsMatchAsync` method for reuse. This enhances code maintainability and readability by reducing redundancy. The new `GetTopic` helper method isolates the topic retrieval logic.

* Add Name property to MassTransitActivityTypeProvider

This commit inserts the Name property in the returned object within the MassTransitActivityTypeProvider class. It ensures that the typeName is included, providing a clearer definition of the activity type.

* Add handling for deleted bookmarks and refactor bookmark removal

Added a new event handler for `BookmarksDeleted` to ensure removed bookmarks are processed correctly. Refactored the bookmark removal logic into a helper method to reduce code duplication and streamline the workflow.

* Switch to asynchronous bookmark queue processing

Refactored the `TriggerWorkflows` handler to use `IBookmarkQueue` instead of directly invoking the `IBookmarkResumer`. This change aims to improve scalability by queueing bookmark resumption requests, enabling better load distribution and async processing. Added necessary helpers and configuration options to support this functionality.

* Remove unused IBookmarkResumer dependency

Simplify the constructor by removing the unused IBookmarkResumer dependency. This cleanup reduces potential confusion and improves code maintainability without impacting functionality.

* Add support for local message processing

Introduced an `IsLocal` property to `MessageReceivedStimulus` for determining if the message event is local to a specific workflow instance. Updated `BookmarkBinding` and related handler methods to utilize `CorrelationId` for local event matching. Removed unused `CorrelatingFields` from `MessageReceived` activity.

* Add nullability checks to IWorker retrieval methods

Updated `GetWorker` methods to return nullable `IWorker` to handle cases where a worker might not exist. Modified code to include null checks and conditional operations to prevent potential null reference exceptions when accessing worker methods.

* Add filtering based on activity type name for triggers and bookmarks

This commit introduces filtering for triggers and bookmarks based on the `MessageReceived` activity type name. It also adds an option to mark messages as local in the `SendMessage` activity, where local messages are delivered to the current workflow instance only. These changes help enhance the management and targeted delivery of messages within the workflow framework.

* Add new Kafka topics and clean up producers config

New topics "topic-3" and "topic-4" were added to the Kafka settings. Unused topic references were removed from the producers configuration to simplify and improve clarity.

* Add predicate to KafkaConsumerActivity in ConsumerWorkflow

Introduced a predicate to the KafkaConsumerActivity using JavaScript expressions to filter messages based on OrderId. This ensures only relevant messages are processed in the workflow.
2024-11-18 13:42:54 +01:00
.github Add workflow to close stale issues and PRs 2024-11-12 11:19:09 +01:00
.nuke Upgrade to NUKE 8.1 (#5952) 2024-09-12 09:56:06 -07:00
build Enable CPM transitive pinning and resolve version conflicts (#6069) 2024-10-31 09:12:21 +01:00
design Add new Elsa logo image 2024-10-03 11:16:14 +02:00
docker Add Elsa.Kafka Module for Kafka Integration with Message Sending and Receiving Activities (#6108) 2024-11-18 13:42:54 +01:00
docs Create bounty-announcement-template.md 2024-07-04 10:27:32 +02:00
gen Update DSL and sample 2022-03-28 13:25:45 +02:00
scripts Add multitenancy support for background tasks (#6059) 2024-10-28 19:38:24 +01:00
src Add Elsa.Kafka Module for Kafka Integration with Message Sending and Receiving Activities (#6108) 2024-11-18 13:42:54 +01:00
test Fix all broken summary xml docs (#6123) 2024-11-18 13:41:27 +01:00
.editorconfig Lazy loading of subgraphs (#5643) 2024-06-25 09:21:50 +02:00
.gitignore Add NUKE build support (#4815) 2024-01-21 10:53:07 +01:00
build.cmd Upgrade to NUKE 8.1 (#5952) 2024-09-12 09:56:06 -07:00
build.ps1 Upgrade to NUKE 8.1 (#5952) 2024-09-12 09:56:06 -07:00
build.sh Upgrade to NUKE 8.1 (#5952) 2024-09-12 09:56:06 -07:00
CONTRIBUTING.md Create CONTRIBUTING.md 2024-01-03 16:22:03 +01:00
Directory.Build.props Integrate improved workflow handling in bulk dispatch tests 2024-10-23 19:12:10 +02:00
Directory.Packages.props Add Elsa.Kafka Module for Kafka Integration with Message Sending and Receiving Activities (#6108) 2024-11-18 13:42:54 +01:00
Elsa.sln Add Elsa.Kafka Module for Kafka Integration with Message Sending and Receiving Activities (#6108) 2024-11-18 13:42:54 +01:00
Elsa.sln.DotSettings Merge remote-tracking branch 'origin/blueberry' into enh/multi-tenancy-refactorings 2024-10-11 19:45:40 +02:00
icon.png Debug 2022-12-20 12:47:41 +01:00
LICENSE Add sources from experimental repo 2022-01-04 09:42:12 +01:00
NuGet.Config Remove obsolete HttpContextExtensions reference and update NuGet URL 2024-11-12 11:04:51 +01:00
README.md Introducing Elsa Guru on Gurubase.io (#6118) 2024-11-13 20:21:45 +01:00

ELSA 3.0

Elsa Workflows

Elsa 3 Prerelease Nuget (with prereleases) feedz.io Docker Image Version (latest semver) Discord Stack Overflow questions Gurubase

For Elsa 2 Click Here

Introduction

Elsa is a powerful workflow library that enables workflow execution within any .NET application. Elsa allows you to define workflows in various ways, including:

  • Writing C# code
  • Using a visual designer
  • Specifying workflows in JSON

Elsa ships with a powerful visual designer

Try with Docker

To give the Elsa Studio + Elsa Server a quick spin, you can run the following command to start the Elsa Docker container:

docker pull elsaworkflows/elsa-server-and-studio-v3:latest
docker run -t -i -e ASPNETCORE_ENVIRONMENT='Development' -e HTTP_PORTS=8080 -e HTTP__BASEURL=http://localhost:13000 -p 13000:8080 elsaworkflows/elsa-server-and-studio-v3:latest

This Docker image is based on a reference ASP.NET application that hosts both the workflow server and designer and is not intended for production use.

By default, you can access http://localhost:13000 and log in with:

  Username: admin
  Password: password

Table of Contents

Documentation

For comprehensive documentation and to get started with Elsa, please visit the Elsa Documentation Website.

Known Issues and Limitations

Elsa is continually evolving, and while it offers powerful capabilities, there are some known limitations and ongoing work:

  • Documentation is still a work in progress.
  • The designer is not yet fully embeddable in other applications; this feature is planned for a future release.
  • C# and Python expressions are not yet fully tested.
  • Bulk Dispatch Workflows is a new activity and not yet fully tested.
  • Input/Output is not yet implemented in the Workflow Instance Viewer.
  • Starting workflows from the designer is currently supported only for workflows that do not require input and do not start with a trigger; this is planned for a future release.
  • The designer currently only supports Flowchart activities. Support for Sequence and StateMachine activities is planned for a future release.
  • UI input validation is not yet implemented.

Features

Elsa offers a wide range of features for building and executing workflows, including:

  • Execution of workflows in any .NET application with support for .NET 6 and beyond.
  • Support for both short-running and long-running workflows.
  • A programming model loosely inspired by Windows Workflow Foundation.
  • A web-based drag & drop designer with support for custom activities.
  • Native support for activity composition, including activities like Sequence, Flowchart, and ForEach.
  • Parallel execution of activities.
  • Built-in activities for common scenarios, such as sending emails, making HTTP calls, scheduling tasks, sending and receiving messages, and more.
  • Workflow versioning and migration via API.
  • Easy integration with external applications via HTTP, message queues, and more.
  • Actor model for increased workflow throughput.
  • Dynamic expressions with support for C#, JavaScript, Python, and Liquid.
  • Persistence agnostic, with support for Entity Framework Core, MongoDB, and Dapper out of the box.
  • Elsa Studio: a modular Blazor dashboard app for managing and designing workflows.

Roadmap

The following features are planned for future releases of Elsa:

  • Multi-tenancy
  • State Machine activity
  • Designer support for Sequence activity & StateMachine activity
  • BPMN 2.0 support
  • DMN support
  • Workflow migration to new versions via UI
  • Capsules ("hot" deployable workflow packages containing activities and configuration)

Use Cases

Elsa can be used in a variety of scenarios, including:

  • Long-running workflows such as order fulfillment and product approval.
  • Short-running workflows such as sending emails and generating PDFs.
  • Scheduled workflows such as sending daily reports.
  • Event-driven workflows such as sending welcome emails when a user signs up.

Programmatic Workflows

Elsa allows you to define workflows in code using C#. The following example shows how to receive HTTP requests and send an email in response:

public class SendEmailWorkflow : WorkflowBase
{
    protected override void Build(IWorkflowBuilder builder)
    {
        builder.Root = new Sequence
        {
            Activities =
            {
                new HttpEndpoint
                {
                    Path = new("/send-email"),
                    SupportedMethods = new(new[] { HttpMethods.Post }),
                    CanStartWorkflow = true
                },
                new SendEmail
                {
                    From = new("alic@acme.com"),
                    To = new(new[]{ "bob@acme.com" }),
                    Subject = new("Your workflow has been triggered!"),
                    Body = new("Hello!")
                }
            }
        };
    }
}

Designed Workflows

Elsa allows you to define workflows using a visual designer. The following example shows how to receive HTTP requests and send an email in response:

Elsa ships with a powerful visual designer

Contributing

We welcome contributions from the community and are pleased that you are interested in helping to improve the Elsa Workflow project! Here are the steps to contribute to our project:

1. Fork and Clone the Repo

To get started, you'll need to fork the repository to your own GitHub account. You can do this by navigating to the Elsa Workflow GitHub repository and clicking the "Fork" button in the top-right corner of the page. Once you have forked the repo, you can clone it to your local machine using the following command:

git clone https://github.com/YOUR_USERNAME/elsa-core.git

Replace YOUR_USERNAME with your GitHub username. For more information on forking a repo, check out the GitHub documentation here.

Incorporating the details about the "apps" folder and its projects into the second point about opening the Elsa.sln using your favorite IDE, we can expand the instructions to guide developers on where to start and what projects they might want to explore first. Here's an updated version of that section with the additional information:

2. Open Elsa.sln Using Your Favorite IDE

After cloning the repository, navigate to the cloned directory and open the Elsa.sln solution file with your preferred IDE that supports .NET development, such as Visual Studio, JetBrains Rider, or Visual Studio Code with the appropriate extensions.

Within the solution, you will find an "apps" folder containing three projects designed to help you get started and explore the capabilities of Elsa Workflow:

  • Elsa.Server.Web: This project is a reference ASP.NET Core application that acts as a workflow server. It's a great starting point if you want to understand how Elsa functions as a server-side workflow engine.

  • Elsa.ServerAndStudio.Web: This project serves a dual purpose. Like Elsa.Server.Web, it acts as a workflow server. Additionally, it hosts the Elsa Studio Blazor WebAssembly app. This is the perfect project to run if you want to see the full capabilities of Elsa, including both the server aspects and the client-side studio experience in one application.

  • Elsa.Studio.Web: This project is a reference Blazor WebAssembly application that solely hosts the Elsa Studio Blazor WebAssembly app. It requires a running Elsa server application to connect to. Use this project if you're interested in focusing on the Elsa Studio UI and its interactions with an Elsa workflow server.

3. Submit a PR with Your Changes

Once you have made your changes, commit them and push them back to your fork. Then, navigate to the original Elsa Workflow repository and create a new Pull Request. Ensure your PR description clearly describes the changes and any relevant information that will help the reviewers understand your contributions. For a detailed guide on creating a pull request, visit Creating a pull request from a fork.

4. Open an Issue First

Before you start working on your changes or submit a pull request, please open an issue to discuss what you would like to do. This step is crucial as it ensures you don't spend time working on something that might not align with the project's goals or might already be under development by someone else. You can open an issue here.

This approach helps us streamline contributions and ensures that your efforts are aligned with the project's needs and priorities. We look forward to your contributions and are here to support you throughout the process. Thank you for contributing to the Elsa Workflow project!

Support

There are various ways to get support for Elsa Workflows, ranging from community-driven channels to enterprise-level services.

Community Support

Elsa has an active and helpful community where you can find support through multiple channels:

  • GitHub Issues for bug reports and feature requests.
  • GitHub Discussions for open-ended conversations, questions, and community-driven support.
  • Discord for real-time support and interaction with the Elsa community.
  • StackOverflow for searching or asking technical questions.

Enterprise Support

For organizations requiring professional support and long-term commitment, ELSA-X offers enterprise-level services and ensures continuity and future development of the Elsa framework, provides custom solutions, and develops commercial extensions tailored to enterprise needs.