* Add retry mechanism for distributed locks with transient error handling and logging - Introduced Polly-based retry pipeline for distributed lock acquisition in `DistributedWorkflowClient` to handle transient errors such as network issues or database connection failures. - Added detailed logging for retry attempts and lock release errors. - Updated project dependencies to include Polly. * Refactor transient exception handling to shared resilience module. Migrated transient exception detection logic from scheduling module to a new shared resilience module. Updated services, jobs, and features to utilize the centralized `ITransientExceptionDetectionService`. This change improves maintainability and promotes reusability across modules. * Add unit tests for transient exception detection and resilience strategy evaluation. - Introduced comprehensive unit tests for `DefaultTransientExceptionDetector`, `ResilienceStrategyCatalog`, `ResilienceStrategyConfigEvaluator`, and `TransientExceptionDetectionService`. - Added helper classes and test data factories to facilitate reusable test patterns for resilience modules. - Updated solution to include `Elsa.Resilience.Core.UnitTests` project. * Add component tests for distributed lock resilience - Introduced new tests to verify retry behavior during transient lock acquisition and release failures. - Added `TestDistributedLockProvider` and related mocks for simulating transient failures. - Updated `WorkflowServer` test services to support the new distributed lock test scenarios. * Refactor distributed lock resilience tests - Consolidated test logic: streamlined test providers, injected services, and reusable test patterns. - Simplified `TestDistributedLockProvider` implementation with enhanced initialization and failure simulation. - Reorganized tests for transient acquisition/release failures to use parameterized `Theory` for improved maintainability. * Refactor transient exception handling: rename interfaces and classes for consistency, update references across codebase, and improve code readability. * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Simplify WorkflowServer setup and DistributedLockResilienceTests by replacing IDistributedLockProvider with TestDistributedLockProvider. * Remove unused `using` directives in unit tests to improve code cleanliness. * Remove `TransientExceptionTypes` helper and inline its usage in tests for improved maintainability. * Add descriptive `DisplayName` attributes to unit tests for improved test clarity. * Fix redundant exception checking in TransientExceptionDetector (#7162) * Initial plan * Fix redundant exception checking in TransientExceptionDetector Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Extract MaxRetryAttempts constant in DistributedLockResilienceTests (#7164) * Initial plan * Extract MaxRetryAttempts constant to eliminate hardcoded magic number Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Make TestDistributedLockProvider thread-safe with Interlocked operations (#7163) * Initial plan * Make TestDistributedLockProvider thread-safe using Interlocked operations Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Update src/modules/Elsa.Workflows.Runtime.Distributed/Services/DistributedWorkflowClient.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/component/Elsa.Workflows.ComponentTests/Scenarios/DistributedLockResilience/DistributedLockResilienceTests.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/modules/Elsa.Workflows.Runtime.Distributed/Services/DistributedWorkflowClient.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/modules/Elsa.Resilience.Core/Services/DefaultTransientExceptionStrategy.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Include `CancellationToken` in distributed lock handling methods for improved cancellation support. * Initial plan (#7168) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> * Cache detector list in TransientExceptionDetector to avoid repeated allocations (#7167) * Initial plan * Cache detector list in field to avoid repeated allocations Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Use IReadOnlyList instead of List for better intent expression Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Fix TestDistributedLockProvider registration to properly decorate IDistributedLockProvider (#7166) * Initial plan * Fix TestDistributedLockProvider registration to use Decorate pattern and fix variable reference bug Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Add runtime check for TestDistributedLockProvider registration Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Refactor `DistributedWorkflowClient` to simplify `Lazy<ResiliencePipeline>` initialization. * Add integration tests for DistributedWorkflowClient lock resilience (#7165) * Initial plan * Fix compilation error: use correct parameter name transientExceptionDetector Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Add integration tests for DistributedWorkflowClient lock resilience - Add SimpleWorkflow for testing distributed lock scenarios - Add tests exercising RunInstanceAsync with transient lock failures - Verify retry logic works correctly with actual workflow execution - Test both acquisition and release failure scenarios - Decorate IDistributedLockProvider to use TestDistributedLockProvider Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Address code review feedback - Add explanatory comment for TestDistributedLockProvider cast - Remove unnecessary blank line for consistent formatting Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> Co-authored-by: Sipke Schoorstra <sipkeschoorstra@outlook.com> * Simplify transient exception strategy by refactoring message pattern matching logic. * Enhance distributed lock mock to support per-lock failure configuration and improve resilience tests. * Refactor `TestDistributedLockProvider` to streamline failure handling logic and improve code clarity. * Remove unused methods and redundant test case from `DistributedLockResilienceTests`. * Refactor `DistributedLockResilienceTests` to simplify workflow client creation, consolidate assertion logic, and remove redundant test cases. * Format `ResilienceStrategyCatalogTests` by removing redundant line breaks in test setup. * Refactor `TransientExceptionDetectorTests` to simplify test setup, consolidate test cases, and remove redundant logic. * Handle `InvalidOperationException` in `XunitLogger` to suppress logging errors during inactive tests. * Update workflows to use .NET 10 and adjust resilience tests project configuration. * Refactor distributed runtime feature and integrations to improve resilience handling, configure services fluently, and add cancellation safeguards in background services. * Update `base_version` to `3.7.0` in GitHub workflow configuration. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com>
88 lines
3.3 KiB
C#
88 lines
3.3 KiB
C#
using System.Net.Sockets;
|
|
|
|
namespace Elsa.Resilience.Core.UnitTests;
|
|
|
|
public class DefaultTransientExceptionStrategyTests
|
|
{
|
|
private readonly DefaultTransientExceptionStrategy _strategy = new();
|
|
|
|
public static TheoryData<Type> TransientExceptionTypes =>
|
|
[
|
|
typeof(HttpRequestException),
|
|
typeof(TimeoutException),
|
|
typeof(TaskCanceledException),
|
|
typeof(IOException),
|
|
typeof(SocketException),
|
|
typeof(EndOfStreamException)
|
|
];
|
|
|
|
public static TheoryData<string> TransientMessagePatterns =>
|
|
[
|
|
"timeout",
|
|
"timed out",
|
|
"connection reset",
|
|
"connection refused",
|
|
"broken pipe",
|
|
"network",
|
|
"end of stream",
|
|
"attempted to read past the end",
|
|
"the connection is closed",
|
|
"connection is not open",
|
|
"failed to connect",
|
|
"no connection could be made",
|
|
"an existing connection was forcibly closed",
|
|
"TIMEOUT",
|
|
"Connection Reset"
|
|
];
|
|
|
|
public static TheoryData<string> NonTransientMessagePatterns =>
|
|
[
|
|
"Some random error",
|
|
"Invalid operation",
|
|
"Null reference"
|
|
];
|
|
|
|
[Theory(DisplayName = "Known transient exception types should be detected as transient")]
|
|
[MemberData(nameof(TransientExceptionTypes))]
|
|
public void IsTransient_KnownTransientExceptionType_ReturnsTrue(Type exceptionType)
|
|
{
|
|
var exception = (Exception)Activator.CreateInstance(exceptionType)!;
|
|
Assert.True(_strategy.IsTransient(exception));
|
|
}
|
|
|
|
[Theory(DisplayName = "Exceptions with transient message patterns should be detected as transient")]
|
|
[MemberData(nameof(TransientMessagePatterns))]
|
|
public void IsTransient_ExceptionWithTransientMessagePattern_ReturnsTrue(string message)
|
|
{
|
|
var exception = new Exception(message);
|
|
Assert.True(_strategy.IsTransient(exception));
|
|
}
|
|
|
|
[Theory(DisplayName = "Exceptions with non-transient messages should not be detected as transient")]
|
|
[MemberData(nameof(NonTransientMessagePatterns))]
|
|
public void IsTransient_ExceptionWithNonTransientMessage_ReturnsFalse(string message)
|
|
{
|
|
var exception = new Exception(message);
|
|
Assert.False(_strategy.IsTransient(exception));
|
|
}
|
|
|
|
[Theory(DisplayName = "Non-transient exception types should not be detected as transient regardless of message")]
|
|
[InlineData(typeof(InvalidOperationException), "Some error")]
|
|
[InlineData(typeof(ArgumentException), "Invalid argument")]
|
|
[InlineData(typeof(NullReferenceException), "Object reference not set")]
|
|
public void IsTransient_NonTransientExceptionType_ReturnsFalse(Type exceptionType, string message)
|
|
{
|
|
var exception = (Exception)Activator.CreateInstance(exceptionType, message)!;
|
|
Assert.False(_strategy.IsTransient(exception));
|
|
}
|
|
|
|
[Theory(DisplayName = "Exceptions with empty messages should be detected based on type only")]
|
|
[InlineData(typeof(TimeoutException), true)]
|
|
[InlineData(typeof(InvalidOperationException), false)]
|
|
public void IsTransient_ExceptionWithEmptyMessage_ChecksTypeOnly(Type exceptionType, bool expectedResult)
|
|
{
|
|
var exception = (Exception)Activator.CreateInstance(exceptionType, "")!;
|
|
Assert.Equal(expectedResult, _strategy.IsTransient(exception));
|
|
}
|
|
}
|