* Add retry mechanism for distributed locks with transient error handling and logging - Introduced Polly-based retry pipeline for distributed lock acquisition in `DistributedWorkflowClient` to handle transient errors such as network issues or database connection failures. - Added detailed logging for retry attempts and lock release errors. - Updated project dependencies to include Polly. * Refactor transient exception handling to shared resilience module. Migrated transient exception detection logic from scheduling module to a new shared resilience module. Updated services, jobs, and features to utilize the centralized `ITransientExceptionDetectionService`. This change improves maintainability and promotes reusability across modules. * Add unit tests for transient exception detection and resilience strategy evaluation. - Introduced comprehensive unit tests for `DefaultTransientExceptionDetector`, `ResilienceStrategyCatalog`, `ResilienceStrategyConfigEvaluator`, and `TransientExceptionDetectionService`. - Added helper classes and test data factories to facilitate reusable test patterns for resilience modules. - Updated solution to include `Elsa.Resilience.Core.UnitTests` project. * Add component tests for distributed lock resilience - Introduced new tests to verify retry behavior during transient lock acquisition and release failures. - Added `TestDistributedLockProvider` and related mocks for simulating transient failures. - Updated `WorkflowServer` test services to support the new distributed lock test scenarios. * Refactor distributed lock resilience tests - Consolidated test logic: streamlined test providers, injected services, and reusable test patterns. - Simplified `TestDistributedLockProvider` implementation with enhanced initialization and failure simulation. - Reorganized tests for transient acquisition/release failures to use parameterized `Theory` for improved maintainability. * Refactor transient exception handling: rename interfaces and classes for consistency, update references across codebase, and improve code readability. * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Simplify WorkflowServer setup and DistributedLockResilienceTests by replacing IDistributedLockProvider with TestDistributedLockProvider. * Remove unused `using` directives in unit tests to improve code cleanliness. * Remove `TransientExceptionTypes` helper and inline its usage in tests for improved maintainability. * Add descriptive `DisplayName` attributes to unit tests for improved test clarity. * Fix redundant exception checking in TransientExceptionDetector (#7162) * Initial plan * Fix redundant exception checking in TransientExceptionDetector Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Extract MaxRetryAttempts constant in DistributedLockResilienceTests (#7164) * Initial plan * Extract MaxRetryAttempts constant to eliminate hardcoded magic number Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Make TestDistributedLockProvider thread-safe with Interlocked operations (#7163) * Initial plan * Make TestDistributedLockProvider thread-safe using Interlocked operations Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Update src/modules/Elsa.Workflows.Runtime.Distributed/Services/DistributedWorkflowClient.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/component/Elsa.Workflows.ComponentTests/Scenarios/DistributedLockResilience/DistributedLockResilienceTests.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/modules/Elsa.Workflows.Runtime.Distributed/Services/DistributedWorkflowClient.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/modules/Elsa.Resilience.Core/Services/DefaultTransientExceptionStrategy.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Include `CancellationToken` in distributed lock handling methods for improved cancellation support. * Initial plan (#7168) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> * Cache detector list in TransientExceptionDetector to avoid repeated allocations (#7167) * Initial plan * Cache detector list in field to avoid repeated allocations Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Use IReadOnlyList instead of List for better intent expression Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Fix TestDistributedLockProvider registration to properly decorate IDistributedLockProvider (#7166) * Initial plan * Fix TestDistributedLockProvider registration to use Decorate pattern and fix variable reference bug Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Add runtime check for TestDistributedLockProvider registration Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Refactor `DistributedWorkflowClient` to simplify `Lazy<ResiliencePipeline>` initialization. * Add integration tests for DistributedWorkflowClient lock resilience (#7165) * Initial plan * Fix compilation error: use correct parameter name transientExceptionDetector Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Add integration tests for DistributedWorkflowClient lock resilience - Add SimpleWorkflow for testing distributed lock scenarios - Add tests exercising RunInstanceAsync with transient lock failures - Verify retry logic works correctly with actual workflow execution - Test both acquisition and release failure scenarios - Decorate IDistributedLockProvider to use TestDistributedLockProvider Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> * Address code review feedback - Add explanatory comment for TestDistributedLockProvider cast - Remove unnecessary blank line for consistent formatting Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com> Co-authored-by: Sipke Schoorstra <sipkeschoorstra@outlook.com> * Simplify transient exception strategy by refactoring message pattern matching logic. * Enhance distributed lock mock to support per-lock failure configuration and improve resilience tests. * Refactor `TestDistributedLockProvider` to streamline failure handling logic and improve code clarity. * Remove unused methods and redundant test case from `DistributedLockResilienceTests`. * Refactor `DistributedLockResilienceTests` to simplify workflow client creation, consolidate assertion logic, and remove redundant test cases. * Format `ResilienceStrategyCatalogTests` by removing redundant line breaks in test setup. * Refactor `TransientExceptionDetectorTests` to simplify test setup, consolidate test cases, and remove redundant logic. * Handle `InvalidOperationException` in `XunitLogger` to suppress logging errors during inactive tests. * Update workflows to use .NET 10 and adjust resilience tests project configuration. * Refactor distributed runtime feature and integrations to improve resilience handling, configure services fluently, and add cancellation safeguards in background services. * Update `base_version` to `3.7.0` in GitHub workflow configuration. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfmskywalker <938393+sfmskywalker@users.noreply.github.com>
152 lines
5.9 KiB
C#
152 lines
5.9 KiB
C#
using NSubstitute;
|
|
|
|
namespace Elsa.Resilience.Core.UnitTests;
|
|
|
|
public class TransientExceptionDetectorTests
|
|
{
|
|
[Fact(DisplayName = "Service with no registered strategies should return false for any exception")]
|
|
public void IsTransient_NoDetectors_ReturnsFalse()
|
|
{
|
|
var detector = CreateDetector();
|
|
var exception = new Exception("test");
|
|
|
|
Assert.False(detector.IsTransient(exception));
|
|
}
|
|
|
|
[Fact(DisplayName = "Service should return true when any strategy detects the exception as transient")]
|
|
public void IsTransient_DetectorReturnsTrue_ReturnsTrue()
|
|
{
|
|
var exception = new Exception("test");
|
|
var strategy = CreateStrategy((exception, true));
|
|
var detector = CreateDetector(strategy);
|
|
|
|
Assert.True(detector.IsTransient(exception));
|
|
}
|
|
|
|
[Fact(DisplayName = "Service with multiple strategies should return true if any one detects as transient")]
|
|
public void IsTransient_MultipleDetectorsOneReturnsTrue_ReturnsTrue()
|
|
{
|
|
var exception = new Exception("test");
|
|
var strategy1 = CreateStrategy((exception, false));
|
|
var strategy2 = CreateStrategy((exception, true));
|
|
var detector = CreateDetector(strategy1, strategy2);
|
|
|
|
Assert.True(detector.IsTransient(exception));
|
|
}
|
|
|
|
[Fact(DisplayName = "Service should return false when all strategies detect the exception as non-transient")]
|
|
public void IsTransient_AllDetectorsReturnFalse_ReturnsFalse()
|
|
{
|
|
var exception = new Exception("test");
|
|
var strategy1 = Substitute.For<ITransientExceptionStrategy>();
|
|
var strategy2 = Substitute.For<ITransientExceptionStrategy>();
|
|
strategy1.IsTransient(Arg.Any<Exception>()).Returns(false);
|
|
strategy2.IsTransient(Arg.Any<Exception>()).Returns(false);
|
|
var detector = CreateDetector(strategy1, strategy2);
|
|
|
|
Assert.False(detector.IsTransient(exception));
|
|
}
|
|
|
|
[Theory(DisplayName = "Service should walk the inner exception chain to find transient exceptions")]
|
|
[MemberData(nameof(InnerExceptionChainTestCases))]
|
|
public void IsTransient_InnerExceptionChainHasTransient_ReturnsTrue(Exception exception, Exception transientException)
|
|
{
|
|
var strategy = Substitute.For<ITransientExceptionStrategy>();
|
|
strategy.IsTransient(transientException).Returns(true);
|
|
strategy.IsTransient(Arg.Is<Exception>(e => e != transientException)).Returns(false);
|
|
var detector = CreateDetector(strategy);
|
|
|
|
Assert.True(detector.IsTransient(exception));
|
|
}
|
|
|
|
[Theory(DisplayName = "Service should inspect AggregateException inner exceptions")]
|
|
[MemberData(nameof(AggregateExceptionTestCases))]
|
|
public void IsTransient_AggregateException_ChecksInnerExceptions(
|
|
AggregateException aggregateException,
|
|
Action<ITransientExceptionStrategy> configureDetector,
|
|
bool expectedResult)
|
|
{
|
|
var strategy = Substitute.For<ITransientExceptionStrategy>();
|
|
configureDetector(strategy);
|
|
var detector = CreateDetector(strategy);
|
|
|
|
Assert.Equal(expectedResult, detector.IsTransient(aggregateException));
|
|
}
|
|
|
|
[Fact(DisplayName = "AggregateException with mixed inner exceptions should be transient if any inner is transient")]
|
|
public void IsTransient_AggregateExceptionWithMultipleInnerOneTransient_ReturnsTrue()
|
|
{
|
|
var transientException = new TimeoutException("timeout");
|
|
var nonTransientException = new InvalidOperationException("invalid");
|
|
var aggregateException = new AggregateException("aggregate", nonTransientException, transientException);
|
|
|
|
var strategy = CreateStrategy(
|
|
(aggregateException, false),
|
|
(nonTransientException, false),
|
|
(transientException, true));
|
|
var detector = CreateDetector(strategy);
|
|
|
|
Assert.True(detector.IsTransient(aggregateException));
|
|
}
|
|
|
|
public static IEnumerable<object[]> InnerExceptionChainTestCases
|
|
{
|
|
get
|
|
{
|
|
var transientException = new TimeoutException("timeout");
|
|
|
|
yield return
|
|
[
|
|
new Exception("outer", transientException),
|
|
transientException
|
|
];
|
|
|
|
yield return
|
|
[
|
|
new Exception("outer", new("middle", transientException)),
|
|
transientException
|
|
];
|
|
}
|
|
}
|
|
|
|
public static IEnumerable<object[]> AggregateExceptionTestCases
|
|
{
|
|
get
|
|
{
|
|
// Scenario 1: One transient inner exception
|
|
var timeoutException = new TimeoutException("timeout");
|
|
yield return
|
|
[
|
|
new AggregateException("aggregate", timeoutException),
|
|
(Action<ITransientExceptionStrategy>)(detector =>
|
|
{
|
|
detector.IsTransient(Arg.Is<TimeoutException>(e => e.Message == "timeout")).Returns(true);
|
|
detector.IsTransient(Arg.Is<Exception>(e => e.GetType() != typeof(TimeoutException))).Returns(false);
|
|
}),
|
|
true
|
|
];
|
|
|
|
// Scenario 2: No transient inner exceptions
|
|
yield return
|
|
[
|
|
new AggregateException("aggregate", new InvalidOperationException("error1"), new ArgumentException("error2")),
|
|
(Action<ITransientExceptionStrategy>)(detector =>
|
|
{
|
|
detector.IsTransient(Arg.Any<Exception>()).Returns(false);
|
|
}),
|
|
false
|
|
];
|
|
}
|
|
}
|
|
|
|
private static ITransientExceptionStrategy CreateStrategy(params (Exception exception, bool isTransient)[] behaviors)
|
|
{
|
|
var detector = Substitute.For<ITransientExceptionStrategy>();
|
|
foreach (var (exception, isTransient) in behaviors)
|
|
detector.IsTransient(exception).Returns(isTransient);
|
|
return detector;
|
|
}
|
|
|
|
private static TransientExceptionDetector CreateDetector(params ITransientExceptionStrategy[] detectors) => new(detectors);
|
|
}
|