▲T3 TRADE · SYSTEM ATLAS
Failure stories

Five fixes and three failures

Each guard on this page exists because an earlier behavior failed. This page tells eight of the stories behind them: five fixes from the RC02 through RC06 review cycle, each verified against its commit and read back out of the code that shipped it, and three war stories preserved as warnings inside the repository itself. The same failure hides inside all of them: an outcome was called a success before anything confirmed it.

submitCancel execution service POST /exchange transport 200 acknowledgement? readCancelAcknowledgement acknowledged per-order success only unconfirmed typed inspect_failed
5
case files, RC02 through RC06, every stat below re-verified with git show --stat before it was printed
3,416
lines inserted across the five fix commits: 325 + 849 + 727 + 720 + 795
347ms
after its own fill that a mission was accused of acting on the exchange directly, now covered by a 60-second grace window apps/server/src/trading/HyperliquidReconciler.ts:994-1004
30s
after which a control press with no durable result stops waiting and reads as interrupted apps/web/src/components/trading/useMissionControls.ts:69

How to read a case file

Each of the five fixes below is told the same way, because incidents are only comparable when the record is uniform. Every field is written from the repository: the symptom from the code the commit replaced, the cause from the comments the fix left behind, the evidence from the files at their current line numbers. The RC identifiers and commit hashes are the review cycle's own; run git show <hash> in the checkout to audit any card.

Symptom
What a user or a log would have seen. The observable lie: a cancel reported gone, a revoke still spinning, a tab trading on the wrong environment.
Root cause
The mistaken assumption. The one sentence the old code believed that was not true, usually a boundary crossed: transport taken for agreement, dispatch taken for done.
The fix
Files and behavior as they ship now. What was added or threaded through, with the current line ranges to read.
The lesson
One plain sentence. The transferable rule, stripped of this codebase's specifics.
Evidence
Citations you can check. Paths, line ranges, and the commit hash with its verified file and insertion counts.

Five fixes in review order

Five fixes, one per card, in the order the review cycle found them. The thread connecting them is invisible in any single card and obvious across all five: each fix moves an unconfirmed outcome one step further from being mistaken for a confirmed one. The Execution page shows the submit path these fixes hardened; this page shows what went wrong first.

RC02

A cancellation succeeds only on an exchange acknowledgement

commit c996478ef · 5 files changed · +325 -9
Symptom
An order could stay alive on the book while every surface reported it cancelled. apps/server/src/trading/HyperliquidExecutionService.ts awaited the exchange POST and discarded the response body, so an application-level rejection arriving over a successful transport was reported as a successful cancellation. The commit's own words: the path "logged 'hit the wire'" instead of failing typed (commit c996478ef).
Root cause
Transport success was read as cancellation success. The response reader deliberately tolerates a status-less cancel envelope for other purposes, and nothing between the POST and the caller distinguished an explicit per-order success from anything else (packages/hyperliquid/src/ExchangeResponse.ts:221-237).
The fix
A strict decoder, readCancelAcknowledgement, acknowledges only an explicit per-order success on a cancel-shaped envelope. Action-level rejection, per-order error, malformed envelope, missing statuses, wrong type, wrong cardinality, and order-status words such as resting or filled all stay unconfirmed, carrying the reason the response actually gave (packages/hyperliquid/src/ExchangeResponse.ts:248-287). HyperliquidExecutionService.ts:995-1007 now inspects the acknowledgement and fails typed as inspect_failed; success means confirmed by the exchange, and even a "not found" reply stays unconfirmed, because only an exchange-authoritative order lookup can prove absence; matching response text cannot (packages/hyperliquid/src/ExchangeResponse.ts:235-237).
The lesson
A successful request proves the request arrived, not that the exchange agreed to it.
Evidence
packages/hyperliquid/src/ExchangeResponse.ts:221-287apps/server/src/trading/HyperliquidExecutionService.ts:995-1012commit c996478ef (5 files, +325)
RC03

Preserve cancellation and block-write uncertainty in every outcome

commit 8ef85f7b3 · 11 files changed · +849 -96
Symptom
An emergency close could flatten a position and report only the flat. Two facts vanished with it: the mission-block write had not confirmed, and some increasing-order cancels had not confirmed either. The code's own framing of the gap RC03 closed: "a close that flattened the position while its entry cancellations or its block write stayed unconfirmed is a different outcome than a clean one, and callers must not parse the notice to tell them apart" (apps/server/src/trading/TradingEmergencyCloseService.ts:69-73).
Root cause
The uncertainty lived only in prose. Outcomes carried a notice string, so any caller that skipped or skimmed it lost the facts, and nothing in the types forced them forward into what the operator reads.
The fix
CancellationReport { acknowledged, unconfirmed }: the shared best-effort cancel pass returns only cloids the exchange explicitly confirmed in acknowledged, and every failed or unconfirmable cancel rides in unconfirmed with its reason, logged at warn per cloid (apps/server/src/trading/RestingIncreasingOrders.ts:36-50, :78-107). The emergency-close outcome union now carries blockWriteConfirmed and unconfirmedCancellations on all three branches, flat, open, and unknown (apps/server/src/trading/TradingEmergencyCloseService.ts:75-117). The exhaustion path warns that unconfirmed entries "are the entries that may reopen exposure under a mission the operator sees as blocked" (apps/server/src/trading/TradingMissionReactor.ts:2057-2068).
The lesson
If a fact would change what the operator does next, it belongs in the outcome type, not in a log line.
Evidence
apps/server/src/trading/RestingIncreasingOrders.ts:35-107apps/server/src/trading/TradingEmergencyCloseService.ts:63-117apps/server/src/trading/TradingMissionReactor.ts:2057-2068commit 8ef85f7b3 (11 files, +849)
RC04

Emergency close results survive all three reactor callers

commit 6e6203dcd · 4 files changed · +727 -12, of which 548 are one new test file
Symptom
The reactor escalates to an emergency close from three places: protection could not be confirmed (apps/server/src/trading/TradingMissionReactor.ts:1399), a stop replacement left the position uncovered (:1600), and the protection watchdog could not re-place a removed stop (:2547). Each call site assembled its own rendering of the outcome, and one of the three got it wrong; nothing pinned the three together (the commit exists to make results "survive all three reactor callers", commit 6e6203dcd).
Root cause
Three hand-rolled renderings of one outcome type. Any future caller would have been a fourth chance to drift, and the worst renderings are the ones nobody re-reads during an escalation.
The fix
One shared rendering: describeEmergencyCloseOutcome is now the only way an emergency-close result becomes text. Flat says flat, open carries the signed remaining size, unknown carries no number, and the RC03 warnings ride along even on a flat outcome (apps/server/src/trading/TradingEmergencyCloseService.ts:119-133). All three callers route through recordEmergencyClose, which uses it (apps/server/src/trading/TradingMissionReactor.ts:365), and 548 lines of new reactor tests pin all three callers in place (commit 6e6203dcd, apps/server/src/trading/TradingMissionReactor.test.ts).
The lesson
When every caller must render an outcome the same way, give them one function and test the callers, not just the function.
Evidence
apps/server/src/trading/TradingEmergencyCloseService.ts:119-133apps/server/src/trading/TradingMissionReactor.ts:365,1399,1600,2547commit 6e6203dcd (4 files, +727)
RC05

Latch the trading environment destination once and keep it recoverable

commit 3224f1861 · 7 files changed · +720 -98, of which 418 are one new routing test
Symptom
The Trade tab traded on whatever project sorted first. The old derivation, projects[0], answered "which environment does the Trade tab trade on?" with "whatever project sorts first", in the module's own words, "which is not a choice anyone made" (apps/web/src/components/trading/tradingEnvironmentSelection.ts:5-8). A sort order or list change mid-session silently moved real, signed orders onto a different environment.
Root cause
The destination was continuously derived from live inputs instead of chosen once, so it re-derived whenever its inputs moved, and no layer treated that movement as the routing change it was.
The fix
A session-scoped store latches the destination exactly once, when the environment catalog first becomes ready: the valid primary, else the sole entry, else none, and several entries with no valid primary require an explicit choice before any trading query or control runs (apps/web/src/components/trading/tradingEnvironmentSelection.ts:10-24, :70-78). Later catalog, project, and primary changes never move it. A destination that vanishes renders unavailable with an explicit Use <label> (<id>) recovery action and never a silent fallback (apps/web/src/components/trading/TradingEnvironmentSelector.tsx:37-65). The pre-latch window renders "Loading trading environments…" and mounts no environment-bound queries (apps/web/src/components/trading/TradeHomePanel.tsx:219). A 418-line routing test pins the state machine (commit 3224f1861, apps/web/src/components/trading/tradingEnvironmentRouting.test.tsx).
The lesson
A routing decision that spends capital must be a choice someone made, latched once, and loud when it stops being valid.
Evidence
apps/web/src/components/trading/tradingEnvironmentSelection.ts:5-148apps/web/src/components/trading/TradingEnvironmentSelector.tsx:37-65commit 3224f1861 (7 files, +720)
RC06

Durable, operator-visible risk-control outcomes end to end

commit f4d78e890 · 15 files changed · +795 -37
Symptom
A §14.7 control press (pause, reduce, close, revoke) dispatched a command, and the outcome of the reactor's exchange work "lived only in server logs" (apps/server/src/persistence/Migrations/094_TradingControlResults.ts:9-12). Worse on the client: void send() discarded the dispatch promise, "so a revoke refused by the domain looked exactly like a revoke still in flight" (apps/web/src/components/trading/useMissionControls.ts:42-47).
Root cause
The accepted dispatch proved only that the request landed. Nothing closed the loop from the reactor's exchange work back to the operator's screen, and the projection the screen reads was pull-only on a slow cadence, with no refusal channel of its own.
The fix
The reactor's recordControlResult writes an append-only trading_control_results row (the control, a completed/failed/unknown status, the composed summary, per-market facts as JSON, and the sequence of the request event it answers) and dispatches a trading.mission.control-result event as the doorbell (apps/server/src/trading/TradingMissionReactor.ts:1641-1700; table at apps/server/src/persistence/Migrations/094_TradingControlResults.ts:25-39). On the web, resolveControlOutcome correlates a result to the press that caused it: same control, at or after dispatch minus a 1 second clock-skew allowance, and a press whose result never lands reads as interrupted after CONTROL_RESULT_TIMEOUT_MILLIS = 30_000 (apps/web/src/components/trading/useMissionControls.ts:69, :89-124). A refresh on the way out of the dispatch lands a successful control in under a second despite the 3 second projection cadence (apps/web/src/components/trading/useMissionControls.ts:150-157). The durability rule is stated in the migration: "the exchange stays the truth; this is the record of what T3 Trade's control did about it" (apps/server/src/persistence/Migrations/094_TradingControlResults.ts:17-21).
The lesson
A control is not done when the command is accepted; it is done when its durable outcome is on the screen, and a refused control must look refused.
Evidence
apps/server/src/persistence/Migrations/094_TradingControlResults.ts:4-39apps/server/src/trading/TradingMissionReactor.ts:1636-1700apps/web/src/components/trading/useMissionControls.ts:42-157commit f4d78e890 (15 files, +795)

Three war stories the code still tells

Not every lesson became a case file. Three incidents remain documented in runtime comments and constants. Each one explains a rule that otherwise looks arbitrary: a grace window, a retirement sweep, a refusal to delete.

The reconciler accused a mission of stealing from itself

The drift detector exists to catch outside interference: position size moved with no fill T3 placed to explain it. One live mission's own entry, five fills all carrying its own cloid, was reported to the model as "someone acted on the exchange directly" 347 ms after it filled (apps/server/src/trading/HyperliquidReconciler.ts:994-996).

Two clocks caused it. traded_at is the exchange's time, observed_at is ours, and observation always trails trade, so a pass could stamp its baseline after the only fills that could explain the next delta, and the next pass saw the position appear from nowhere (apps/server/src/trading/HyperliquidReconciler.ts:988-993). The fix is the 60-second attribution grace window, ATTRIBUTION_GRACE_MILLIS = 60_000: a fill up to 60 seconds before the previous observation still counts as explaining a change (apps/server/src/trading/HyperliquidReconciler.ts:1004). External fills carry no cloid, so the attribution query never explains them at any window width. The code names the one tradeoff honestly: an external order placed within the grace of one of the mission's own fills can be masked, "which is a far cheaper mistake than crying theft at the mission's own entry" (apps/server/src/trading/HyperliquidReconciler.ts:998-1003). The Reconciliation page shows the loop this window lives inside.

Four profit targets fired three hours later, at the wrong trade

Position-scoped watches used to outlive their position. On the mission this was measured on, four profit targets armed for the first long fired three hours later against an unrelated one, and two targets armed for a short fired on a long (apps/server/src/trading/TradingMissionReactor.ts:2076-2079). The cause is the predicate itself: a pnl_above watch "asks the position what it is worth and does not ask which position" (apps/server/src/trading/TradingMissionReactor.ts:2079-2080).

So position-scoped watches are retired the moment the mission goes flat, and the retirement is wired into both ways flat: the watchdog stop-out path and the agent-runtime-driven close path, which the earlier wiring missed, so "six closes in a row therefore retired nothing" (apps/server/src/trading/TradingMissionReactor.ts:2082-2088). The Missions watches wakes page covers the watch lifecycle this rule bounds.

The boot sweep that erased a running soak

The boot sweep's job is to withdraw authority from missions whose thread was deleted while the server was down. An earlier version deleted those missions outright and read orphanhood off projection_threads, a rebuildable projection table; the first projection rebuild made every live mission look orphaned, and the sweep erased a running soak (apps/server/src/trading/TradingMissionSweep.ts:9-14).

The sweep now revokes from durable tables and deletes nothing. Mission rows are the permanent record of what was traded, and revoking "is safe even if the orphan test is ever wrong again" (apps/server/src/trading/TradingMissionSweep.ts:9-14). A mission still holding a position is never touched: it is reported as stranded, because revoking it would leave real exposure nobody is authorized to manage (apps/server/src/trading/TradingMissionSweep.ts:16-20, :67-71).

One lesson, learned five times

Read in sequence, the five case files show the same mistake at five boundaries. RC02 found it at the exchange boundary: a transport success was being called a cancellation. RC03 pushed the distinction into the outcome types, so unconfirmed facts ride along instead of vanishing. RC04 pinned the rendering across every caller, so three escalation paths cannot drift apart again. RC05 found the same shape in a choice: a derived environment was a destination nobody confirmed, so the fix latches it once. RC06 found it in the operator's own controls: an accepted dispatch was being treated as a done control, and a refused one was invisible.

The Safety invariants page states the rule in the large; this page is where clauses of it earned their place. The Risk control page covers the §14.7 controls RC06 made durable, and the Web & desktop page covers the surfaces RC05 made explicit. The sentence underneath all five fixes is short enough to memorize: never call an unconfirmed outcome a success.

Go deeper

Every file below was opened at its current line numbers before this page quoted it. The commits are in the checkout's history; the counts in each card come from git show --stat run against them.

Cancellation truth

  • packages/hyperliquid/src/ExchangeResponse.ts:221-287 the strict decoder and its contract
  • apps/server/src/trading/HyperliquidExecutionService.ts:976-1012 submitCancel inspecting the acknowledgement
  • apps/server/src/trading/RestingIncreasingOrders.ts:35-107 the shared cancel pass and its report

Outcomes and callers

  • apps/server/src/trading/TradingEmergencyCloseService.ts:63-133 the three-way outcome and the one rendering
  • apps/server/src/trading/TradingMissionReactor.ts:1399,1600,2547 the three escalation call sites
  • apps/server/src/trading/TradingMissionReactor.ts:2057-2068 the exhaustion warning that names the risk

The operator loop

  • apps/server/src/persistence/Migrations/094_TradingControlResults.ts:4-39 the append-only table and its durability rule
  • apps/server/src/trading/TradingMissionReactor.ts:1636-1700 recordControlResult and the doorbell event
  • apps/web/src/components/trading/useMissionControls.ts:42-157 press-to-result correlation and the 30-second timeout

War story sites

  • apps/server/src/trading/HyperliquidReconciler.ts:985-1013 the 347 ms incident and the grace window
  • apps/server/src/trading/TradingMissionReactor.ts:2073-2088 why position-scoped watches retire on flat
  • apps/server/src/trading/TradingMissionSweep.ts:9-48 why the sweep revokes and never deletes