▲T3 TRADE · SYSTEM ATLAS
T3 Trade System Atlas · Reconciliation

The exchange is authoritative.

One service converges T3 Trade's local tables to what Hyperliquid testnet actually holds. Eight triggers run it, one pass does the work, and when the two sides disagree, local state loses.

triggers §18.2 · 8 entry points canonical reads account · orders · fills diff vs previous snapshot persist migration-037 tables doorbell tradingAccountInvalidations periodic backstop · every 5 s while a position is open
8
named triggers, each a named entry point into one reconcile call HyperliquidReconciler.ts:67-76
5 s
periodic backstop while a position is open TradingFillReconciler.ts:195
60 s
attribution grace before a position change is called external HyperliquidReconciler.ts:1004
15 s
protection window before escalation to emergency close protection.ts:143-151

The rule

Two writers share one account. The agent signs orders through T3 Trade, and anything else that can reach the wallet, including a human with the exchange UI open, moves the same position. T3 Trade splits the work by authority: local state records T3 Trade's decisions and actions, and the exchange is authoritative for positions, orders, and fills.

Local fills, position snapshots, and open orders remain provisional until exchange-authoritative state confirms them HyperliquidReconciler.ts:5-9. This page calls that confirmed state canonical state. Local state never outranks Hyperliquid HyperliquidReconciler.ts:1117-1118, and the persist step repeats the rule HyperliquidReconciler.ts:1150-1152.

The rule brackets every mutation. The reactor reconciles before execution so the preview checklist and the loss budget gate see reconciled truth, not a stale local cache TradingMissionReactor.ts:1888-1895, and again after the signed order lands, because local records are hints until that pass confirms them TradingMissionReactor.ts:2018-2022. What happens between the two reconciles is the Execution story.

What the overwrite never touches

Three columns survive every pass because the exchange does not report them. peak_unrealised_pnl only ever ratchets up while a position is held, trough_unrealised_pnl only ratchets down, and opened_at is stamped once and held HyperliquidReconciler.ts:411-428.

They are the memory the closing self-review reads: how far a winner came off its best, how far offside the trade went, and how long it took. The upsert keeps them through MAX, MIN, and COALESCE on the conflict path HyperliquidReconciler.ts:476-478.

Eight named triggers

Each trigger is a named entry point that funnels into one reconcile call, recording which trigger fired HyperliquidReconciler.ts:129-146. TradingFillReconciler owns the live three, marked amber below: the fills subscription, the reconnect convergence, and the periodic backstop TradingFillReconciler.ts:1-17.

server_startup
The mission follow loop, when a booting server adopts live missions TradingMissionReactor.ts:2354
● websocket_reconnect
The fill reconciler, on every WebSocket reconnect. Replayed frames cannot be trusted against canonical state, so it re-reads through the Info API; the reconciler is idempotent, so a redundant read is safe TradingFillReconciler.ts:161-167
before_execution
The mission reactor, right before guards and the budget read state for a submit TradingMissionReactor.ts:1888-1895
after_submission
The mission reactor, right after the signed order landed TradingMissionReactor.ts:2018-2022
● after_fill
The fill reconciler: each userFills WebSocket frame carrying at least one fill on the mission's market TradingFillReconciler.ts:145-156
after_position_update
User controls, the exhaustion guard, and emergency close converge after they move exposure, so the snapshot they leave behind is true even if everything else is down TradingControlService.ts:360 TradingExecutionGuard.ts:252 TradingEmergencyCloseService.ts:385
before_resuming_paused_mission
The reactor, before a paused mission is allowed to run again TradingMissionReactor.ts:1053
● periodic_while_position_open
The fill reconciler, every 5 seconds while the canonical position is non-flat, as a backstop for any frame the socket missed TradingFillReconciler.ts:169-195

The periodic gate also reads local belief before it skips a flat mission. If T3's tables still hold a position, or an execution record sits in a non-terminal status, the pass runs anyway: while T3 thinks it holds something, one more reconcile is always worth it, and a record parked at accepted or submitted on a flat mission otherwise held its risk reservation until the next server start TradingFillReconciler.ts:72-100.

One pass, start to finish

reconcile is the single convergence path. Every trigger enters the same function and runs the same steps HyperliquidReconciler.ts:1100-1269.

baseline

Baseline first, before anything overwrites it

The previous snapshot row is read at the top of the pass because two different things depend on it: the external-change diff needs the old size, and the closing self-review needs the excursion columns this pass is about to clear HyperliquidReconciler.ts:845-862. The previous account value is read the same way HyperliquidReconciler.ts:965-972.

canonical reads

Exchange-authoritative reads, in parallel

Account, open orders, and fills are fetched together with Effect.all at unbounded concurrency HyperliquidReconciler.ts:1119-1126. The open-orders read is the full account-wide set, not only T3's own rows: a linked TP/SL child carries a cloid T3 never chose, and it still protects the position HyperliquidReconciler.ts:108-113 HyperliquidReconciler.ts:214-221. The persisted subset is narrowed later, by cloid and by market HyperliquidReconciler.ts:242-274.

Fills are floored at the mission's start time, because userFills is an account-wide read: without the floor, a brand-new thread adopted the previous thread's positions and receipts HyperliquidReconciler.ts:306-317.

derive

Derive what only the exchange can confirm

The mark price is arithmetic, not a second call. The clearinghouse position payload carries no mark field, so the pass derives it from the unrealised PnL the exchange already reports: mark = entry + upnl / size, sign carried by size HyperliquidReconciler.ts:186-192.

protectedSize is confirmed, not inferred: confirmedProtectedSize sums the qualifying resting reduce-only triggers from the canonical order set and clamps to the position HyperliquidReconciler.ts:1129-1145 packages/trading-contracts/src/protection.ts:240-259. A grouped response, a local record, and an untriggered child all read as protection if you ask the wrong source; only this read counts.

persist

Persist, overwrite, heal

The position snapshot upserts, fills append idempotently, pre-mission fills an earlier build adopted are dropped, and open orders are replaced with the canonical set HyperliquidReconciler.ts:1149-1158. Fill identity is the exchange's per-trade tid, never the action hash: every partial of one order shares one hash, and keying on it once collapsed a dozen partials into a single row that reported a fully-filled order at a fraction of its size HyperliquidReconciler.ts:276-304.

settle

Settle executions, release reservations

Abandoned and accepted executions are settled against the canonical orders, then the risk reservations of terminal records release in the same pass, ordered so a record settled here has its reservation released with it HyperliquidReconciler.ts:1161-1181.

doorbell

Narrate, then ring the doorbell

External changes are classified and queued to the inbox HyperliquidReconciler.ts:1188-1195, fills on server-rested take-profit orders are matched against the protection ledger HyperliquidReconciler.ts:930-963, and the one pass that first sees the position gone writes the closing review HyperliquidReconciler.ts:1200-1205.

The pass ends by ringing accountProjection.invalidate with the trigger name as its reason HyperliquidReconciler.ts:1259. Quiet periodic passes log at debug, because twelve identical lines a minute had made the mission log unreadable HyperliquidReconciler.ts:1241-1256.

The drift taxonomy

A pass compares the canonical present against the previous snapshot. Five kinds of drift have names, inbox events, and in most cases a scar that put them there.

External position change external_close · external_reduce · external_increase

Position size moved with no attributed fill since the previous pass. The classifier names it by what it did to the exposure HyperliquidReconciler.ts:813-826, and a reversal counts as an increase: crossing through flat opens new exposure in the other direction, which is the thing that needs a stop and a decision.

The cloid identifies who submitted the order. Every order T3 Trade signs carries a deterministic cloid, while an order placed in the Hyperliquid UI carries none. A fill with a cloid came from T3 Trade, including a triggered exchange-native reduce-only stop. A fill without one came from another source HyperliquidReconciler.ts:1006-1024.

4 → 0external_close
4 → 2external_reduce
4 → -2external_increase (reversal)

Before this event existed, closing a position in the exchange UI only cleared the snapshot. The mission's agent runtime, called the harness in code, continued managing a position that no longer existed HyperliquidReconciler.ts:1026-1033.

External transfer external_transfer

Account value moved more than $1 between two passes while the mission was flat on both sides HyperliquidReconciler.ts:828-836 HyperliquidReconciler.ts:1062-1077. Fees and funding move it by cents; a deposit or a withdrawal moves it by dollars. With a position open the value tracks unrealised PnL continuously, so a diff means nothing and is never consulted.

The message the mission receives keeps the priorities straight: your mandate is unchanged; what you can afford is not HyperliquidReconciler.ts:1070-1077.

Closed trade trade_closed

The one pass that first observes a held position gone writes the closing self-review, durable first, then queued to the inbox where the next wakeup carries it HyperliquidReconciler.ts:864-914. The deduplication key is the trade's opening time. It used to be the observation instant, which differs by milliseconds between passes: one live close was announced to the harness twice, carrying the same scorecard both times HyperliquidReconciler.ts:870-874 HyperliquidReconciler.ts:900.

Server-rested take-profit fills take_profit_filled

The take-profit reconcile rests a reduce-only ALO at the plan's target, and nothing told the harness when it filled. On the mission this was found on, two rungs filled between wakes, the position quietly halved, and the model attributed the drop to the give-back trigger it had armed itself HyperliquidReconciler.ts:916-928.

The reconciler now matches each fill against the protection ledger and queues it with the fill's own id as the dedup key, so re-observing the same fill on the next pass cannot queue it twice HyperliquidReconciler.ts:930-963. The target itself has since become a wake, not an order; see Missions watches wakes.

Protection gap protection_lost

The position is larger than the confirmed protected size. A partial fill, a scale-in, and a parent cancellation that took its children with it all end in this same state, which is why one routine converges all three TradingProtectionService.ts:14-19. The reactor's watchdog family runs every 5 seconds, lease-gated, and its protection guard queues a protection_lost event naming the exposed size TradingMissionReactor.ts:2450-2527 TradingMissionReactor.ts:2886-2920.

place sized stop → confirm from canonical → cancel stale

The repair runs place-then-cancel; overlapping stops are harmless because reduce-only orders cannot open exposure TradingProtectionService.ts:29-36.

Sixty seconds of grace, one 347 ms scar

The attribution window compares two clocks. traded_at is the exchange's trade time and observed_at is ours, and observation always trails trade. A pass that reads the account after the fills have already traded stamps a baseline later than the only fills that could explain the next pass's delta, and the next pass then sees the position appear from nowhere.

That is not hypothetical. One mission's own entry, five fills all carrying its own cloid, was reported to the model as "someone acted on the exchange directly" 347 ms after it filled. The fix widens the attribution lookback to 60 seconds HyperliquidReconciler.ts:985-1004.

A genuinely external action produces fills without a cloid, so widening the lookup window does not make those fills look internal. The window could mask an external order placed within 60 seconds of one of T3 Trade's own fills. The code accepts that limited risk to avoid misclassifying T3 Trade's entry as external HyperliquidReconciler.ts:998-1002.

Repair: overwrite, then protect

For positions, orders, and fills, repair is the overwrite itself: the pass replaces local rows with canonical truth and settles what no longer holds. The one repair that needs sequencing is protection.

A grouped submission proves nothing. The linked TP/SL child that went out with an entry may be untriggered, rejected, or already cancelled, so coverage is read back from frontendOpenOrders every time TradingProtectionService.ts:20-23.

When a gap is found, the correctly sized stop is placed and confirmed before the stale one is dropped. That order is safe and the reverse is not: overlapping reduce-only protection is harmless because reduce-only orders cannot open exposure, while a gap between the cancel and the placement is exactly the window the invariant forbids TradingProtectionService.ts:29-36.

The window is tight on purpose: 15 seconds total, a canonical re-read every 750 ms, at most 3 placement attempts packages/trading-contracts/src/protection.ts:143-151. The contract's own comment calls the window a liability, not a budget. More convergence scars are unpacked on the Failure stories page.

When the window closes uncovered

  1. The 5-second watchdog finds size beyond confirmed protection and queues protection_lost with the exposed size TradingMissionReactor.ts:2450-2527.
  2. It re-places the stop using epoch seconds as the execution sequence, so the watchdog's cloid can never collide with a harness sequence, which are small counters TradingMissionReactor.ts:2529-2536.
  3. If the outcome is escalate, the bounded emergency close takes over: block increases, cancel non-reduce-only orders, read fresh canonical state, send a reduce-only IOC, reconcile, retry with fresh reads, at most 3 attempts TradingMissionReactor.ts:2542-2551 TradingEmergencyCloseService.ts:14-33 TradingEmergencyCloseService.ts:51.
  4. Either way the harness wakes with cause order_updated: its stop was pulled out from under it TradingMissionReactor.ts:2562-2563.

Why this design

This is an old distributed systems lesson applied to a wallet. When two writers share state and only one can hold the truth, the other keeps a journal and diffs against the ledger on a schedule. Hyperliquid is the ledger. T3 Trade is the decision journal: the Event spine records what T3 Trade decided, and reconciliation records what the exchange did about it.

That is why reconciliation runs before and after every mutation. User controls also work without the agent runtime: they run when the provider process is unavailable, the session is unreachable, or the model is still responding. When they change exposure, they reconcile with after_position_update TradingControlService.ts:360. Risk control reads the loss budget that reconciled fills feed, and the preview checklist reads the position this pass just wrote.

The pass ends at the UI. The invalidation is one doorbell per pass, not a payload: invalidate publishes a doorbell event and the WebSocket layer serves the view and its subscription off the same instance the reconciler publishes into TradingAccountProjection.ts:12-13 runtimeLayer.ts:331-334. The client watches tradingAccountInvalidations and refetches once per server-side change instead of polling, and the watchlist and alert feed ride the same doorbell packages/client-runtime/src/state/orchestration.ts:89-99. The old 3-second mission poll is gone; a 30-second fallback survives only for older servers or a silently dead subscription apps/web/src/lib/tradingMissionsState.ts:22-32. The Web & desktop page follows the rest of that path.

The rule, stated once. Local state records T3 Trade's decisions and actions. The exchange is authoritative for positions, orders, and fills. Every confirmed exposure increase must have exchange-native protection, and a failure path must never leave a position silently unprotected AGENTS.md (repository working rules, Product boundaries).

Go deeper

The pass itself

  • apps/server/src/trading/HyperliquidReconciler.ts 1,613 lines: triggers at 67-76, the pass at 1100-1269, drift detection at 795-1098
  • apps/server/src/trading/TradingFillReconciler.ts 202 lines: the live three triggers and the wake rule
  • packages/trading-contracts/src/protection.ts confirmedProtectedSize at 240-259, the 15 s window at 143-151

Protection and escalation

  • apps/server/src/trading/TradingProtectionService.ts place-then-cancel rationale at 29-36
  • apps/server/src/trading/TradingMissionReactor.ts watchdog family at 2886-2920, protection guard at 2450-2564
  • apps/server/src/trading/TradingEmergencyCloseService.ts the fixed seven-step order at 14-33, the 3-attempt bound at 51

Consumers of a pass

  • apps/server/src/trading/TradingAccountProjection.ts the doorbell contract at 12-13
  • packages/client-runtime/src/state/orchestration.ts the invalidation subscription at 89-99
  • apps/web/src/lib/tradingMissionsState.ts push with a 30 s fallback at 22-32
  • Next page: Contracts covers the wire types a pass consumes.