Target: hummingbot/hummingbot
Issues: hummingbot/hummingbot#7294 and hummingbot/hummingbot#7295
PR: https://github.com/hummingbot/hummingbot/pull/8306
Field Lab: https://github.com/scarab-systems/scarab-field-lab/tree/main/field-tests/hummingbot-hummingbot-7294
This field test targeted a close-order lifecycle boundary in Hummingbot’s PositionExecutor.
The visible issue was duplicate close-order behavior.
The deeper issue was more specific:
a close order could emit MarketOrderFailureEvent
the executor could clear that failed close order for retry
the same order could later receive a legitimate late fill through the existing order-tracking path
but the executor no longer associated that late fill with the original close order
the shutdown retry loop could then submit another market close
That is a financial execution lifecycle boundary.
The problem was not simply that an order failed.
The problem was that the system could lose the identity needed to reconcile what happened next.
In plain English:
the executor could believe the close order failed, forget it for retry purposes, then fail to recognize that the same order later filled
That opened the door to duplicate close behavior.
Field Lab record
The public case record for this field test is available in the Scarab Field Lab:
https://github.com/scarab-systems/scarab-field-lab/tree/main/field-tests/hummingbot-hummingbot-7294
Upstream posture
This was a clean upstream repair candidate.
The issues described a real lifecycle failure surface.
The repair was narrow.
The patch stayed inside PositionExecutor and its regression coverage.
ClientOrderTracker behavior was intentionally left unchanged.
That matters because the diagnostic finding did not point to the general order tracker as the broken ownership surface.
The boundary was in the executor’s handling of a failed close order that could later fill.
Diagnostic result
This field test examined Hummingbot’s close-order lifecycle behavior.
The useful result was a narrowly scoped boundary finding.
The failure lived across a small but important chain:
close order submission
market-order failure event
failed close-order clearing
shutdown retry behavior
late fill reconciliation
duplicate close prevention
That chain points to an executor-owned boundary.
The correct repair was not to redesign order tracking.
It was not to broadly rewrite shutdown behavior.
It was not to make every failed order persist forever as active state.
It was to preserve enough failed close-order identity for the executor to reconcile a legitimate late fill for the same order.
That is the repair lane.
Failure shape
The failure shape was a failure-then-fill race.
A close order could be treated as failed.
The executor could clear that order so a retry could happen.
But the same order could later be observed as filled through the normal tracking path.
At that point, the executor needed to know that the late fill belonged to the previously failed close order.
Without that preserved identity, the executor could continue down the retry lane and submit another close order.
That creates duplicate close behavior.
In a trading system, that is not just a local state bug.
Close-order identity is part of execution truth.
If the system loses the relationship between the failed order and the later fill, it can become coherent around the wrong execution state.
Boundary
The boundary here is:
failed close-order identity versus late-fill reconciliation inside PositionExecutor
Hummingbot’s order tracker can still report order events.
The executor still has to preserve enough lifecycle truth to interpret those events correctly.
That was the boundary Scarab surfaced.
The question was:
When a close order fails and is cleared for retry, does PositionExecutor still preserve enough identity to recognize that same order if it later fills?
Before the repair, that answer could be no.
That is where the close-order lifecycle drift entered.
What changed
The PR keeps failed close-order identity available long enough for PositionExecutor to reconcile a late fill for the same order.
If the failed close order is later observed as filled, the executor restores it as the close order and avoids placing another close order.
The patch keeps the repair focused on the executor lifecycle boundary.
Changed files:
hummingbot/strategy_v2/executors/position_executor/position_executor.py
test/hummingbot/strategy_v2/executors/position_executor/test_position_executor.py
The patch also adds regression coverage for the failure-then-fill race.
Why this was not an order-tracker rewrite
The tempting repair would be to move the problem into general order tracking.
That would make the patch broader than the actual failure boundary.
The issue was not that ClientOrderTracker could not observe the late fill.
The issue was that PositionExecutor had already cleared the close-order identity it needed in order to reconcile that late fill safely.
So the repair stayed where the lifecycle obligation lived:
inside PositionExecutor.
That is important because the executor is the component deciding whether a new close order should be placed.
The order tracker can provide event truth.
The executor still has to preserve enough local lifecycle truth to act on that event correctly.
Why the diagnostic result mattered
This case is useful because it shows how a system can keep moving while its execution truth has already slipped.
Each piece can look reasonable in isolation:
a market close order fails
the executor clears it for retry
shutdown logic tries to ensure the position closes
the order tracker later observes fills
But the lifecycle agreement between those pieces can break.
The executor needs to know whether the close-order lifecycle has already resolved.
If it forgets the failed order too early, a real late fill can arrive without being reconciled to the original close attempt.
The diagnostic boundary was:
when late execution events and retry state disagree, preserve the smallest identity surface needed to keep the lifecycle truthful
This framing kept the patch small.
It avoided a broad order-tracking rewrite.
It avoided a shutdown-system redesign.
It repaired the executor-owned seam where close-order truth was lost.
Validation
The patch was validated with focused and related test coverage.
Validation passed for:
targeted regression test for the close-order failure-then-fill race
full PositionExecutor test file: 35 tests
ClientOrderTracker tests: 35 tests
The validation supports the targeted claim:
PositionExecutor now has regression coverage for the close-order failure-then-late-fill race that could previously lead to duplicate close behavior.
For the full validation record and public status, see the Field Lab case record:
https://github.com/scarab-systems/scarab-field-lab/tree/main/field-tests/hummingbot-hummingbot-7294
Field test result
This was a bounded close-order lifecycle repair candidate for Hummingbot.
The issue reduced to:
a close order could emit MarketOrderFailureEvent
the executor could clear the failed close order for retry
the same order could later receive a legitimate late fill
the executor no longer associated that late fill with the failed close order
shutdown retry handling could submit another market close
the repair preserved failed close-order identity long enough to reconcile the late fill
a focused regression test was added
related executor and order-tracker tests passed
That is the repair lane.
This patch does not claim to resolve every Hummingbot strategy, connector, shutdown, or order lifecycle issue.
It fixes the specific boundary where failed close-order identity and late-fill reconciliation stopped agreeing.
Why this finance surface matters
This field test is important because it moves Scarab into a financial-style execution surface.
That changes the weight of the diagnostic question.
In a financial system, a late fill is not just an event.
A retry is not just control flow.
A close order is not just an object in memory.
These are pieces of operational truth.
They affect how the system understands positions, execution state, and whether another market action should happen.
That makes the Scarab theory sharper:
software drift is not always visible as a broken build or a failing UI
sometimes drift appears when a system loses the identity needed to know what actually happened
Here, Scarab found the boundary.
The boundary was close-order lifecycle identity.
The failure was a failed close order that could later fill without being reconciled.
The repair restored the executor’s ability to preserve that truth before placing another close order.
That is the diagnostic theory in practice:
Find the boundary.
Preserve the truth.
Repair the drift without expanding the surface unnecessarily.
Public claim
The correct claim for this field test is:
Scarab Diagnostic Suite produced a bounded repair candidate for hummingbot/hummingbot#7294 and hummingbot/hummingbot#7295, where PositionExecutor could clear a failed close order for retry and then fail to reconcile a legitimate late fill for that same order, allowing duplicate close behavior. The upstream PR preserves failed close-order identity long enough to reconcile late fills, avoids placing another close order when the original close later fills, and adds targeted regression coverage. Validation passed through the new regression, the full PositionExecutor test file, and related ClientOrderTracker tests. This does not claim to resolve every Hummingbot strategy, connector, shutdown, or order lifecycle issue; it fixes the close-order lifecycle boundary where failed-order identity and late-fill reconciliation stopped agreeing.
Disclosure: This field report was prepared with AI-assisted editing from my own field-test notes, public issue and PR records, validation summary, and repair record. The technical claims and final wording were reviewed before publication.
Top comments (0)