Resetting a channel also means retiring its consumer.
Celery workers use pidbox messaging for remote-control commands. The worker-side component owns both a messaging channel and a consumer attached to that channel. Under reset conditions, the code closed and recreated the channel, but the existing consumer was not explicitly canceled and cleared first.
Zain Nadeem authored PR #10363 to make that lifecycle explicit. The merged change covers reset, shutdown, repeated cleanup, and an integration scenario that exercises worker control traffic.
A reset rebuilt one resource while leaving its dependent object active.
A consumer is not just a passive field on the pidbox object. It represents an active relationship with the messaging transport. During a reset, closing the underlying channel and constructing a new consumer changes that ownership graph. If the old consumer is not canceled and dereferenced, the object model can retain stale state across repeated cycles.
close channel → create channel → replace consumer referencecancel consumer → clear reference → close channel → recreateThe issue becomes more important under repeated control activity because reset behavior is no longer a one-time exceptional path; it is a lifecycle that must remain stable across multiple iterations.
Model the objects by ownership, not by assignment order alone.
The relevant code in celery/worker/pidbox.py exposed two related state variables: the active channel and its consumer. Reviewing reset and shutdown paths showed that each path performed only part of the cleanup sequence.
The reset path also had to remain compatible with recovery: after retiring the old pair, the worker still needed to create a fresh channel and consumer and continue answering control commands. That made successful recreation part of the fix contract, rather than cleanup being considered complete as soon as cancellation returned.
The investigation identified three invariants worth protecting:
- an existing consumer is canceled before the channel it uses is closed;
- the stored consumer reference is cleared before a replacement can be created;
- calling cleanup again does not cancel the same consumer twice.
Those invariants drove both the implementation order and the regression tests.
Channel cleanup and consumer cleanup were treated as one operation.
The prior code relied on closing the channel as if that also completed the consumer’s lifecycle. At the object level, however, the pidbox instance still held its consumer reference. The reset path then created another consumer for the replacement channel.
The missing step was explicit retirement of the old consumer before this transition. Without it, the code’s local references did not accurately describe which transport objects were still active.
Cancel, clear, then close.
PR #10363 updates the reset cleanup path to cancel the current consumer when present, set the stored reference to None, and only then close the channel. The shutdown path also clears the reference after cancellation.
# Simplified lifecycle
if self.consumer is not None:
self.consumer.cancel()
self.consumer = None
self._close_channel()Clearing the reference immediately after cancellation makes repeated cleanup idempotent with respect to that consumer. A later shutdown or reset sees None rather than attempting a second cancellation on an object that has already been retired.
Protect the call order locally and the worker behavior end to end.
The contribution adds both unit and integration coverage. Unit tests verify cancellation during a reset, replacement of the consumer, and behavior across repeated reset cycles. Shutdown coverage checks that a canceled consumer is cleared so cleanup is not repeated.
The integration test broadcasts the error-triggering control path five times, then verifies that the worker still responds to ping and can execute a task successfully. That sequence checks continued control-plane and task-processing behavior after repeated resets.
| Level | Scenario | Protected behavior |
|---|---|---|
| Unit | Single reset | Old consumer canceled before replacement |
| Unit | Repeated reset | Each lifecycle retires the current consumer |
| Unit | Shutdown after cleanup | No double cancellation through a stale reference |
| Integration | Repeated control errors | Worker remains responsive and processes tasks |
The fix joined object-state correctness with operational coverage.
Zain authored PR #10363 against the behavior reported in issue #10340. The pull request changed the pidbox lifecycle code, added focused unit tests, and introduced an integration task and regression scenario. It was merged on 6 July 2026 as commit 201573d87c619932465618239af19b57cb3ca041.
The result is a narrow cleanup correction: it does not redesign pidbox messaging or change command semantics. It makes resource retirement explicit at the transition where replacement occurs.
Lifecycle code should make ownership transitions observable.
- Closing a parent transport does not make every dependent object’s in-memory lifecycle explicit.
- Clear references immediately after successful cleanup to make repeated shutdown paths safe.
- Test both the local cleanup contract and the operational behavior that depends on it.
- Repeated-reset tests reveal state accumulation problems that a single happy-path test cannot expose.