fix: call task.uncancel() after catching CancelledError in shield loops (Python 3.11+)#1523
Open
brucearctor wants to merge 1 commit into
Open
Conversation
On Python 3.11+, asyncio.Task tracks a cancellation counter via Task.cancelling()/Task.uncancel(). When CancelledError is caught inside a while-True/asyncio.shield loop without calling uncancel(), the counter stays elevated and Python re-throws CancelledError at every subsequent await. This causes: 1. Duplicate commands (e.g. RequestCancelExternalWorkflow) sent to the Temporal server 2. Spurious ERROR-level 'exception in shielded future' log lines from temporalio.worker._workflow_instance The fix adds task.uncancel() (guarded by hasattr for Python <=3.10 compatibility) after each CancelledError catch in all 6 affected shield loops: - run_activity() in _outbound_schedule_activity - run_child() in _outbound_start_child_workflow - start-wait loop in _outbound_start_child_workflow - operation_handle_fn() in _outbound_start_nexus_operation - start-wait loop in _outbound_start_nexus_operation - _signal_external_workflow Fixes temporalio#1504
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Python 3.11+,
asyncio.Tasktracks a cancellation counter viaTask.cancelling()/Task.uncancel(). WhenCancelledErroris caught inside awhile True/asyncio.shieldloop without callinguncancel(), the counter stays elevated and Python re-throwsCancelledErrorat every subsequentawait.This causes:
RequestCancelExternalWorkflow) sent to the Temporal servertemporalio.worker._workflow_instanceFix
Adds
task.uncancel()(guarded byhasattrfor Python ≤3.10 compatibility) after eachCancelledErrorcatch in all 6 affected shield loops:run_activity()in_outbound_schedule_activityrun_child()in_outbound_start_child_workflow_outbound_start_child_workflowoperation_handle_fn()in_outbound_start_nexus_operation_outbound_start_nexus_operation_signal_external_workflowAfter
uncancel(), the cancellation counter returns to 0 so the nextawait asyncio.shield(...)blocks normally until the future resolves, rather than immediately re-raisingCancelledError. Thewhile Trueloop then works as designed — it waits for the result after the cancel has been handled.Fixes #1504