Brokerage connectivity and token resilience

A dedicated control plane that maintains brokerage authorization across many users, connections, accounts and regions — where the governing constraint is not throughput but the guarantee that exactly one party may commit a credential change at any moment.

Active-active platform Active-passive per connection Bounded leases Monotonic fencing Versioned atomic commit Envelope encryption Fail closed on partition

Sanitized. This page describes an engagement in generalized terms. The client, its brokerage relationships, user counts, volumes, commercial terms and internal identifiers are withheld. Integrations appear as Brokerage A through Brokerage E and the behaviour attributed to each is a composite drawn from patterns that occur across the industry — no row describes a named vendor, and none should be read as a statement about one. Every service-level figure on this page is a design objective, never a published result; see the footer.

01 The invariant

Everything else in this reference is downstream of one sentence. It was written before the design and every proposal was checked against it.

At most one current owner may commit a refresh result for a given broker_connection_id and token_version. A stale worker or a stale region may execute a refresh — it cannot be prevented from doing so, because it does not know it is stale — but it must not be able to commit once a higher fencing token exists.

The distinction between executing and committing is the whole design. A partitioned worker cannot be stopped from calling a brokerage; no amount of locking reaches across a network partition to halt a process that believes it is healthy. What can be guaranteed is that its result is rejected at the point of persistence, by the store, on a condition it cannot satisfy. Systems that try to prevent execution build increasingly elaborate locks and still fail. Systems that make the commit conditional need only get one predicate right.

A Why not prevent execution

A worker paused by garbage collection, a hypervisor or a partition wakes believing it holds a valid lease. Its belief is unfalsifiable from inside the process. Prevention has to happen where the truth is.

B Why version alone is insufficient

A compare-and-swap on the version stops a writer working from a stale read. It does not stop a former owner region that read recently and is about to write on authority it has since lost.

C Why the fence is the answer

A monotonically increasing token issued by a single global authority, required on every state-changing commit, means a recovered former owner presents a lower value and is rejected by predicate rather than by courtesy.

02 Canonical hierarchy

The most consequential decision in the platform is which object owns a credential. The two intuitive answers — the user and the account — are both wrong, and both fail only in production.

Object	Meaning	Cardinality	Owns credential
User	Platform identity. Holds no brokerage credential of its own.	1 → many connections	No
Broker connection	One separately authorized brokerage login or grant. The unit of refresh scheduling, locking, regional ownership, token versioning and recovery.	Many per user, including repeated at the same brokerage	Yes
Broker account	A tradable account exposed by a connection.	1 connection → one or many accounts	Inherits
Token set	Encrypted access, refresh and brokerage-specific session material.	Exactly 1 current committed set per connection	—
Broker adapter	Versioned brokerage-specific authorization, refresh, validation and error behaviour.	1 per brokerage per integration version	—

If you scope to the user. A user with two separately authorized logins at the same brokerage has two independent credentials with independent expiries and independent rotation. Scoping refresh to the user serializes work that is genuinely independent and, worse, invites logic that treats the two credentials as interchangeable. They are not.

If you scope to the account. Where a brokerage issues one credential per login covering many accounts, per-account scheduling produces N concurrent refreshes of one rotating credential. This is the original defect, reintroduced by the data model rather than by a race.

Accounts inherit the connection's credential unless a brokerage explicitly issues account-scoped material — which some do. The adapter declares this, and the platform schedules accordingly rather than assuming one shape.

03 Component architecture

The token service is a control plane with a deliberately narrow interface. Nothing outside it reaches credential storage, and it never reaches into the trading path.

Consumers reach the platform only through its API. Credential material never crosses the dashed control-plane boundary in a payload — jobs, queue messages and telemetry carry identifiers, versions and fences only.

Component	Responsibility	Implementation characteristics
Token Service API	The only internal interface: connection registration, usable-token retrieval, refresh coordination, status, reauthorization and revocation.	Stateless instances; authenticated workload identity; strict tenant and service authorization.
Token Orchestrator	Validity calculation, ownership check, lease acquisition, adapter invocation, state transition, commit coordination.	No authoritative in-memory state; idempotent handlers throughout.
Durable Scheduler	Schedules each refresh ahead of expiry using the brokerage's safety buffer plus jitter.	Persistent schedules; payload carries identifiers only; completed one-shots self-delete.
Reconciliation Scanner	Finds overdue, unscheduled, inconsistent and stranded connections.	Runs independently on a short cycle; partitioned and elected safely.
Refresh Queues	Backpressure, redelivery, bounded retry and dead-letter handling.	Regional; partitioned per brokerage; never carry credential values.
Refresh Workers	Perform the fenced refresh and the atomic commit of its result.	Multi-AZ; bounded leases; interruption behaviour defined at every step.
Metadata Store	Source of truth for state, version, expiry, owner region, failure history and audit pointers.	Transactional and strongly consistent on the commit path.
Token Store	Access, refresh and session ciphertext.	Envelope encryption; tightly scoped decrypt; committed atomically with metadata.
Global Ownership	Owner region and monotonic fencing token per connection.	Strongly consistent conditional writes; single-item compare-and-set; no transaction required.
Broker Registry	Capabilities, adapter version, refresh buffer, rate limits, session and egress restrictions, error map.	Versioned configuration under controlled rollout.
Observability & Audit	Metrics, traces, alerts, dashboards, tamper-evident audit evidence.	Redacted at the serialization layer; dimensioned by brokerage, region and connection.

04 Connection state machine

Normalized states, identical across every brokerage. The adapter's job is to map a vendor's vocabulary onto these; the platform reasons about nothing else.

The two transitions that matter operationally are Refreshing → Active on a successful conditional commit, and Refreshing → Reauth Required, which is the only path that reaches the user.

State	Meaning	Exit condition	Trading readiness
Pending	Registered, awaiting the user completing the authorization flow.	Callback validated and first token set committed.	Not ready
Active	A committed token set exists and has not reached its refresh point.	Scheduler reaches `next_refresh_at`.	Ready
Refresh Due	Scheduled or reconciliation-detected work exists; no worker holds it.	A worker proves ownership and acquires a lease.	Ready
Refreshing	A worker holds a bounded lease and is executing the adapter path.	Conditional commit succeeds, fails, or the lease expires.	Ready
Retrying	A retryable failure occurred; bounded backoff is in progress.	Success, retry budget exhausted, or circuit opens.	Ready, at risk
Broker Outage	The brokerage circuit is open. Not this connection's fault and not the user's problem yet.	Circuit half-opens and a probe succeeds.	Ready, at risk
Reauth Required	Confirmed terminal credential failure. Only the user can resolve it.	User completes a new authorization.	Not ready
Suspended	Operator hold — used during migration cohorts and incident containment.	Explicit operator release.	Not ready
Revoked	Terminal. User or brokerage withdrew the grant, or the platform revoked it.	None. A new connection is required.	Not ready

Retrying and Broker Outage are deliberately distinct from Reauth Required. Collapsing them is the most common design error in this domain and it is expensive in both directions: treat every rejection as user-action-required and you train users to re-authorize when nothing was wrong; treat every rejection as retryable and you hammer a brokerage that is already unwell, which in the worst case gets the platform's egress addresses throttled or blocked.

05 The fenced refresh sequence

Nine steps. The behaviour of an interruption at each is defined rather than incidental — which is not the same as safe. Interruption at step 6 is the case the design has to answer for.

#	Required behaviour	If the worker dies here
1	Scheduler or reconciliation emits a job carrying connection identifier, expected token version, owner region, fence and correlation identifier — never credentials.	Queue redelivers after the visibility timeout. No state changed.
2	Worker rejects jobs that are revoked, migrated, already refreshed, or carry a stale fence. Cheap checks first.	Redelivery; rejection is idempotent.
3	Worker proves current regional ownership against the global authority and acquires a bounded connection-level lease.	Lease expires unclaimed. Reconciliation re-emits. No state changed.
4	Worker commits a durable intent record — connection, expected version, fence, correlation identifier, state `IN_FLIGHT` — before contacting the brokerage.	Intent exists with no outcome. This is the record that makes step 6 detectable — it turns a silent divergence into a known-ambiguous state reconciliation can act on. It does not make a lost credential recoverable.
5	Worker decrypts the current committed token under its workload identity and re-checks the version it just read against the version it was dispatched with.	Plaintext dies with the process. Intent remains `IN_FLIGHT`.
6	Adapter performs the refresh under a brokerage-specific timeout, rate limiter and circuit breaker.	The hard case. The brokerage may have rotated. See crash-point analysis.
7	Worker validates the entire response — including any rotated refresh token and every expiry field — before treating it as usable. A malformed response is a terminal failure, not a success with gaps.	Intent remains `IN_FLIGHT`; same recovery as step 6.
8	One transaction commits ciphertext, expiration, incremented version, next schedule and state, conditioned on the expected version and the current fence, and resolves the intent record.	Transaction rolls back. Nothing partial persists. Intent remains `IN_FLIGHT`.
9	Queue acknowledged only after the commit. Temporary failures take bounded retries; confirmed terminal failures transition to Reauth Required.	Message redelivers. Step 2 detects the version has already advanced and discards it.

Acknowledge after commit, never before. Acknowledging on receipt converts every worker crash into silent credential drift. The cost of acknowledging late is a duplicate delivery, which step 2 discards for the price of one indexed read. The cost of acknowledging early is a connection whose stored credential no longer matches the brokerage's, with nothing anywhere recording that it happened.

06 Crash-point analysis

The window between a successful brokerage response and the local commit is the entire problem. Everything else is comparatively ordinary engineering.

C1, C2 and C5 are trivially recoverable. C3 is indistinguishable from C4 and is handled as C4. C4 is the one a design has to answer for.

Point	What is true	Recovery	Residual risk
C1	Lease held, nothing else done.	Lease expires. Reconciliation re-emits the job. The next worker starts from step 1.	None
C2	Intent record exists, brokerage not contacted.	Reconciliation finds the stranded intent, confirms via the adapter that the current credential still validates, and safely retries.	None
C3	Request in flight; unknown whether it reached the brokerage.	Treated identically to C4, because it cannot be distinguished from it. The recovery path is the same and so is the cost.	Same as C4 — indistinguishable
C4	Brokerage rotated; the new credential existed only in the lost process. Stored refresh token may already be dead.	Reconciliation must not retry with the stored value — at a rotating brokerage that is precisely what replay detection catches, and the documented response is to invalidate the token. It calls the adapter's `validate()` first, but validation of the stored access token is necessary and not sufficient: a successful refresh does not invalidate the previously issued access token, so validation succeeding proves only that the old access token has not itself expired — not that rotation failed to take effect. Discrimination requires a brokerage-specific signal declared in `capabilities()`: an introspection or session endpoint reporting the live credential generation, or an issuance timestamp that advanced. Where the adapter declares no such signal and the matrix records rotation-on-refresh, the connection is marked Reauth Required rather than retried into a revocation.	User re-authorization in the rotating case
C5	Commit succeeded; acknowledgement may not have been sent.	Message redelivers. Step 2 sees the version has advanced past the job's expectation and discards it.	None

The instinct to retry is the failure. Rotation exists so that an authorization server can detect the possible replay of a superseded refresh token. It cannot tell a genuine attacker from a legitimate client retrying after its own crash, and it is not required to try: RFC 9700 §4.14.2 puts it plainly — the server "cannot determine which party submitted the invalid refresh token, but it will revoke the active refresh token." Several widely deployed authorization servers go further and invalidate the entire descendant token family. A recovery path that retries blindly converts a local process crash into an outage only the end user can repair.

Concurrency-induced false positives are a documented, recurring bug class — not a theoretical concern. Major identity providers ship an explicit reuse-leeway or grace interval precisely because concurrent refreshes otherwise trip replay detection on legitimate traffic, and public advisories continue to be filed against OAuth client libraries and providers for forking or revoking token families on concurrent redemption. Designing as though a single refresh owner is optional is designing for this bug.

07 Three layers of control

Reviewers consistently proposed removing one. Each covers a failure the others do not.

1 Bounded worker lease

Prevents: two workers in the same region doing redundant work.
Does not prevent: a worker paused past its lease expiry that wakes believing it still holds it. A lease is a coordination hint, not a guarantee.

2 Optimistic version CAS

Prevents: a writer working from a stale read overwriting newer state. The commit is conditional on token_version.
Does not prevent: a former owner region that read recently and still carries a plausible version.

3 Monotonic fencing token

Prevents: any commit from a party whose authority has been superseded, regardless of how fresh its read was.
Does not prevent: wasted execution — which is acceptable, because execution without commit is merely expensive.

Required database controls

Connection identity that is independent of user identity and account identity, and unique.
Optimistic compare-and-swap on token_version as a commit precondition.
The current fencing_token required on every state-changing commit, checked in the predicate — not in application code.
Atomicity across new ciphertext, expiration, version, state and next_refresh_at.
Finite leases, such that an expired owner cannot commit with an obsolete fence.
A monotonic fence_high_water column in the commit store, written as part of ownership transfer before the new owner performs any work, and updated only on a new_fence > current predicate. The commit predicate is then job_fence >= fence_high_water AND token_version = expected.
Audit and attempt metadata containing identifiers and normalized outcomes only.

The fence must be enforced by the store that accepts the commit, not only by the store that issues it. This is the detail most designs miss, and Kleppmann is explicit about it: the storage server has to take an active role in checking tokens and rejecting writes on which the token has gone backwards. If the fence is issued by the global authority and never persisted alongside the commit path, there is a live window — after the former owner's lease expires but before the new owner writes anything — in which a paused former owner commits successfully, because the version compare-and-swap has nothing newer to compare against. Recording the high-water fence in the commit store as the first act of an ownership transfer is what closes it.

On the split between stores. The commit path needs a multi-row transaction; the fence needs a cross-region strongly consistent single-item compare-and-set. Those are different guarantees, and the design deliberately uses a store suited to each rather than compromising one to consolidate. Both support conditional writes evaluated against current state. Neither is asked to do the other's job. The alternative — one store providing both — was evaluated and is recorded, with the conditions that should trigger a revisit, in ADR-05.

08 Brokerage capability matrix

The artifact the whole integration strategy rests on. Vendor documentation supplies the happy path; the columns that decide the design come from controlled testing and observed production behaviour.

Anonymized composite. Brokerages A–E below are not named vendors and are not a one-to-one mapping to the client's integrations. Each row is a behaviour class that occurs in this market. The point of the table is the shape of the variation a platform has to absorb, not a directory of who does what.

Integration	Refresh rotation	Credential scope	Session model	Egress	Design consequence
Brokerage A	Rotates on every use	Per login; accounts inherit	Concurrent clients tolerated	Allowlisted	Highest-risk integration. Single-owner enforcement is mandatory, and the C4 recovery path must validate before retrying. Fixed egress constrains regional expansion.
Brokerage B	Stable refresh token	Per login; accounts inherit	Concurrent clients tolerated	Open	Forgiving. A concurrent refresh is wasteful rather than destructive — which makes it a poor integration to design against, and a good one to pilot on.
Brokerage C	Rotates on every use	Per login	Keepalive required — session lapses on inactivity independently of token validity	Open	Two independent clocks. A valid token is not a live session, so readiness has to reflect both, and the keepalive is scheduled work in its own right.
Brokerage D	Not applicable — long-lived grant	Per login	Exclusive — the platform's session can be displaced by another session established under the same identity	Allowlisted	The single-owner rule extends beyond the platform's own workers to any other holder of the same login, which makes it a product and communications problem as much as an engineering one.
Brokerage E	Rotates, with a short reuse grace interval	Per login; some material account-scoped	Concurrent clients tolerated	Open	The grace interval hides concurrency defects during testing and stops hiding them under load. Treated as if it did not exist.

Dimensions recorded per integration

Authorization

Flow type, PKCE support, callback constraints, consent granularity, whether re-authorization requires human interaction and on what cadence.

Lifetimes

Access and refresh validity, whether a refresh extends the window or the original expiry stands, and the safety buffer the platform schedules against.

Rotation

Whether the refresh token changes on use, whether any reuse grace exists, and the observed response to presenting a superseded value.

Session

Whether a keepalive is required independently of refresh, whether sessions are exclusive per identity, and documented maintenance windows.

Scope

Whether credentials attach to the login or to individual accounts, and how account enumeration behaves after a change.

Limits & egress

Rate limits and their granularity, throttling response semantics, and whether connections must originate from allowlisted addresses.

Errors

The vendor's vocabulary mapped onto the normalized set, with particular attention to which rejections genuinely mean the user must act.

Recovery

Observed behaviour after timeouts, partial responses and mid-rotation failures — the column that can only be filled by testing.

This document decays. Brokerage behaviour changes without notice and the matrix matters most at the moment it is wrong. It is a maintained artifact with a named owner and a review cadence, re-validated by the certification suite on every adapter release — not a discovery deliverable that was accurate once.

09 Adapter contract

Brokerage specifics live behind a versioned interface. The core reasons about normalized outcomes and nothing else.

Operation	Contract	Certification requirement
`capabilities()`	Declares rotation behaviour, scope model, session model, buffers, limits and egress requirements.	Must match observed sandbox behaviour; drift fails the suite.
`authorize()`	Completes the authorization flow and returns a validated initial token set.	State validation, single-use state, redirect allowlist and replay rejection all exercised.
`refresh()`	Exchanges the current credential. Returns the complete new set, or a normalized failure.	Must reject partial responses. Must surface a rotated value distinctly from a reissued one.
`validate()`	Cheaply establishes whether the currently held credential is live, without mutating it.	Must be non-mutating. The C4 recovery path depends on this, and an adapter whose validate consumes a credential is worse than none.
`revoke()`	Terminates the grant at the brokerage and confirms termination.	Idempotent; a second call on an already-revoked grant succeeds.
`mapError()`	Maps the vendor vocabulary onto: retryable, rate-limited, brokerage-unavailable, terminal, user-action-required.	Every documented vendor code mapped explicitly. Unknown codes default to retryable-with-low-budget, never to user-action-required.

Unknown errors default toward caution, not toward the user. Defaulting an unrecognized code to user-action-required means every undocumented vendor hiccup sends a re-authorization prompt to someone whose credential was fine. Defaulting to a low retry budget bounds the cost of being wrong and surfaces the unmapped code in telemetry, where it becomes an adapter change rather than a user's afternoon.

10 Brokerage isolation

One vendor's bad day must not become every vendor's bad day.

Queue partitioning

Each brokerage has its own partition. A backlog at one cannot consume the shared depth that others need, and head-of-line blocking is confined to the integration that caused it.

Rate limiting

Per-brokerage limiters configured beneath the published ceiling, because the published ceiling is where throttling begins, not where it is safe to operate.

Circuit breaking

Per brokerage, with a half-open probe. An open circuit moves connections to Broker Outage — a state that is visible, bounded and does not reach the user.

Concurrency caps

A bulkhead per brokerage so that a slow vendor cannot occupy the whole worker pool. Latency at one integration becomes queue depth there, not starvation everywhere.

Retry budgets

Bounded per brokerage and per connection. Unbounded retry against a struggling vendor is indistinguishable from an attack and is treated as one.

Egress separation

Where allowlisting applies, egress is pinned to pre-allocated addresses the vendor has approved, rather than to an address pool that can expand without notice.

11 Multi-region model

Active-active at the platform level. Active-passive per connection. The two are not in tension — they operate at different granularities.

Level	Operating model	Rationale
Entire platform	Active-active	Every approved region serves API traffic and runs workers continuously. A standby that is never exercised is a standby nobody has proven.
Different users	Any healthy region	No affinity required. Users share no mutable credential state.
Different connections, same user	May run simultaneously in different regions	Connections are genuinely independent. Serializing them would be a self-inflicted bottleneck.
One broker connection	Active-passive — exactly one current owner region	The invariant. Active-active refresh of one rotating credential is not an optimization; it is the original defect.
Accounts under a connection	Inherit connection ownership	Unless the brokerage issues account-scoped material, in which case the adapter declares it and each becomes its own ownership unit.

12 Failover and failback

Failback is the direction that actually loses credentials, because a returning owner holds state it believes is current and is not.

Region 1 is not prevented from executing. It is prevented from committing, by a predicate it cannot satisfy, in the store — where its own belief about itself is irrelevant.

Failover rules

Failover may occur only after the previous lease has expired or been explicitly revoked. The transfer itself is the global authority's grant of a strictly higher fencing token, recorded — in both the authority and the commit store's high-water mark — before the new owner performs any work.
During a control-plane partition, a region without current global authority fails closed for refresh commits. It does not proceed on a stale belief about ownership.
The new owner reads the latest committed token version before contacting the brokerage, never from a cached or replicated view it happened to hold.
A recovered former owner reconciles, discards stale work, and cannot resume as owner automatically under any circumstance.
Failback is a deliberate ownership transfer with a fence increment. It must never copy an older token over newer rotated state.

Fail-closed is a business decision, made in daylight. It costs refreshes during a partition, and some of those become user-visible reauthorizations. It was approved at executive level with that cost stated, precisely because a rule decided during an incident gets reversed during the incident — and the reversal is what causes the damage. The alternative, proceeding on a stale ownership belief, risks the one outcome that cannot be undone.

13 Service objectives

Set at architecture review as constraints the design had to be capable of — not derived afterwards from what the system happened to do.

Objective	Target	Measurement note
Token API availability	≥ 99.99% monthly	Measured at the API boundary on successful usable-token retrieval, not on infrastructure health. Declared regional disaster-recovery events are excluded and reported separately — a single regional recovery consumes several times a month's error budget, so folding the two together would make both meaningless.
Eligible refresh completed before expiration	≥ 99.99%	Denominator excludes connections already in Reauth Required or Revoked. This is the number the business actually feels.
Refresh success, excluding confirmed brokerage or user action	≥ 99.95%	Excludes confirmed brokerage outages and confirmed user-side revocations, both of which are attributable from telemetry.
Availability-zone failure	Automatic, no material interruption	Verified by fault injection rather than asserted from the deployment topology.
Regional recovery time objective	≤ 15 minutes	Full-estate ownership transfer, including fence increments and reconciliation catch-up. Rotating and short-lifetime integrations transfer in the first batch, because a fifteen-minute window exceeds the entire access-token lifetime at some brokerages. Connections whose lifetime is shorter than the transfer window are expected to need a post-transfer catch-up refresh; those windows are excluded from the refresh-before-expiry denominator and reported separately.
Regional recovery point objective	≤ 1 minute	Applies to metadata and audit history — and to credential state on the same terms. An unplanned cross-region failover carries a non-zero recovery point measured in seconds, so a rotation committed inside that window, and the intent record covering it, can both be lost. The conditional commit prevents stale writes; it does not recover lost ones. Connections whose last commit falls inside a lost window are treated as C4 on recovery.

These are this platform's approved targets. They are not benchmarks, not vendor service-level agreements, and not transferable to a different design. A published objective is a commitment about a specific architecture on specific infrastructure; quoting one out of that context is how organizations end up committed to numbers nobody designed for.

14 Security controls

Control	Requirement
Encryption	TLS in transit. Envelope encryption at rest, with separate key boundaries per environment and per region. Platform application credentials in a managed secrets service; high-volume user token material in encrypted database fields — different access patterns and different rotation models, kept apart deliberately.
Least privilege	Only the token service's workload identity and audited break-glass roles may decrypt credential material. There is no routine human path to plaintext.
No propagation	No plaintext credential in queues, events, schedules, logs, traces, metrics, analytics extracts, support tooling or exception reports. Enforced at the serialization layer, not by reviewer discipline — redaction that depends on every future contributor remembering is not a control.
Regional decrypt	A region may decrypt only where a key replica has been deliberately created and that replica's own policy and grants permit it. Replicated ciphertext does not confer the right to use it. Key replication widens blast radius and is treated as an authorization decision rather than a convenience.
Authorization callback	State validation, PKCE where the brokerage supports it, single-use callback state, strict redirect allowlist, replay protection. The callback is attacker-reachable by definition.
Service identity	Authenticated, authorized service-to-service calls on workload identity. No direct database access by trading services — the strongest control in the design is an absent connection string.
Support tooling	Built to operate on identifiers and normalized status. A support user cannot see a credential because the interface has no code path that renders one.
Audit	Tamper-evident events for credential reads, refresh, revoke, reauthorize, ownership transfer, failover and privileged action.
Detection	Synthetic canary credentials planted where a leak would surface, plus automated scanning of logs and traces for credential-shaped material. A control that is never tested is a belief.
Retention	Agreed retention and secure deletion for revoked credentials and operational history, set with the compliance function rather than inherited from a default.
AI boundary	Anomaly detection over telemetry is read-only and welcome. No automated agent may author a token state transition, acquire a lease, decrypt credential material or initiate an ownership transfer. AI-assisted development is permitted with human review and excluded from the concurrency and cryptographic paths, where a plausible-looking change is the most dangerous kind. Recorded in the governance record, not left as a convention.

15 Threat model

Named scenarios, revisited at each delivery gate. A threat model that is not re-read is a document, not a control.

Scenario	Mitigation	Residual
Two workers use the same rotating refresh token	Single-owner rule; bounded lease; version CAS; fence checked in the commit predicate.	Structurally prevented
Worker crashes after brokerage success, before local commit	Durable intent record written before the call; validate-before-retry recovery; no blind retry at rotating brokerages.	Reauthorization in the rotating case
Stale region returns after another obtained a higher fence	Commit rejected on the fence predicate; forced reconciliation; no automatic resumption of ownership.	Structurally prevented
Database failover during rotation	Single-transaction commit; within a region the intent record survives and reconciliation re-establishes state. Across regions, an unplanned failover can lose both the commit and its intent record.	Possible credential divergence in the unplanned cross-region case
Brokerage returns malformed or incomplete token data	Full response validation including every expiry field; partial responses treated as terminal failures, never as partial successes.	Handled
Support user or unrelated service attempts to read plaintext	Decrypt restricted to workload identity and audited break-glass; support tooling has no rendering path; denied decrypts alert.	Break-glass, audited
Brokerage outage or rate limit causes queue growth and cross-brokerage starvation	Per-brokerage partitions, limiters, breakers, bulkheads and retry budgets.	Contained
Authorization callback replayed or redirected to an unapproved endpoint	Single-use state, strict redirect allowlist, PKCE where supported, replay rejection.	Handled
Credential material leaks into telemetry	Serialization-layer redaction; automated log and trace scanning; synthetic canary credentials as the tripwire.	Detective, not preventive
Legacy and new refresh paths both act on one connection during migration	Atomic ownership transfer with a technical interlock — the legacy path is prevented from acting, not merely configured not to.	Structurally prevented

16 Concurrency and failure test matrix

Testing was the deliverable, not the check on it. Failure injection ran against real infrastructure, because the failures that matter are the ones mocks do not model.

ID	Scenario	Pass standard
T-01	Two workers dispatched for one connection simultaneously	Exactly one commit succeeds. The other is rejected on the version predicate and exits without side effects.
T-02	Worker paused beyond lease expiry, then resumed	Commit rejected on the fence. No brokerage call is repeated after rejection.
T-03	Rotation race — two refreshes of one rotating credential forced	Only one reaches the brokerage. Connection remains Active with a single valid credential.
T-04	Scheduler and reconciliation emit for the same connection in the same window	Duplicate work is detected at step 2 and discarded. One commit.
T-05	Worker terminated at each of the five defined crash points	Recovery matches the documented behaviour per point. No connection is left inconsistent.
T-06	Worker terminated between brokerage success and commit, rotating brokerage	Intent record found. Validation runs before any retry. Connection reaches Active or Reauth Required — never a blind retry.
T-07	Full service restart with work in flight	All in-flight work is either completed or re-emitted. Nothing is stranded past one reconciliation cycle.
T-08	Database failover during an open refresh transaction	No partial commit. Intent record survives. Reconciliation resolves within one cycle.
T-09	Queue duplicates a message after acknowledgement	Discarded at step 2 on the advanced version. No second brokerage call.
T-10	Availability-zone loss under sustained load	Automatic recovery with no material interruption and no missed refresh window.
T-11	Clock skew introduced between workers and the store	Lease and expiry decisions remain safe. No commit succeeds on a lease the store considers expired.
T-12	Brokerage returns sustained rate limiting	Circuit opens for that brokerage only. Other integrations show no degradation in queue depth or latency.
T-13	Controlled isolation of the owning region — connectivity severed, not the region destroyed	Ownership transfers with a fence increment inside the recovery objective. No stale commit, and no credential lost outside the documented cross-region recovery-point window.
T-14	Failed region recovers and attempts to resume ownership	All commits rejected on the fence. Region reconciles and discards. It does not become owner without a deliberate transfer.
T-15	Control-plane partition — region cannot reach the global authority	Region fails closed for refresh commits. No commit proceeds on a stale ownership belief. Alert fires.
T-16	Deliberate failback to the recovered region	Transfer is explicit, fence increments, and no older credential state overwrites a newer rotation.
T-17	Unauthorized service attempts to read the credential store directly	Denied at the identity boundary. Denial alerts. No network path exists to attempt it from the trading tier.
T-18	Canary credential planted in a code path that logs liberally	Detected by automated scanning within the agreed window. Alert routes to security, not to the owning team alone.
T-19	Authorization callback replayed, and separately redirected off-allowlist	Both rejected. Neither produces a usable token set. Both are audited.
T-20	Decrypt attempted from a region holding replicated ciphertext but no grant	Denied by the replica's own key policy. Denial is visible in audit.
T-21	Legacy refresher attempts to act on a migrated connection	Technically prevented, not merely inactive. Attempt is recorded.
T-22	Migration cohort rolled back mid-flight	Ownership returns cleanly. No connection is left owned by both systems or by neither.
T-23	Cohort held through a complete expiry cycle before the next proceeds	Every connection in the cohort refreshes at least once under the new platform before promotion.

17 Observability

Built as part of the platform, not assembled for the readiness review. Every dimension is redacted by design.

Signal	Required visibility	Response
Expiry risk	Connections expiring in 5, 15 and 30 minutes, and — the part that matters — whether a viable refresh or retry path currently exists for each.	Page if the viable-path count diverges from the expiring count.
Queue health	Depth, oldest message age, redelivery rate, stranded jobs and dead-letter count, by brokerage and region.	Alert per brokerage; a single-vendor backlog is not a platform incident.
Refresh quality	Success rate, latency distribution, attempts, retries, terminal failures, completion-before-expiry.	Anomaly detection over the per-brokerage baseline, not a static threshold.
Ownership	Lease acquisition failures, fence increments, regional moves, and any multiple-owner anomaly.	A multiple-owner anomaly pages immediately. It should be impossible.
Brokerage health	Timeouts, throttling responses, server errors and normalized circuit state per brokerage.	Circuit state drives the Broker Outage transition and the status surface.
Security	Denied decrypts, unauthorized service calls, leak indicators, break-glass access.	Routed to security independently of the owning team.
Reauthorization	Rate and cause distribution, split by whether the platform or the brokerage originated the terminal state.	A rise in platform-originated reauthorization is a defect signal, not a user-behaviour signal.

The most valuable single metric is platform-originated reauthorization rate. Brokerage-originated reauthorization is normal and outside the platform's control. Platform-originated reauthorization means the system asked a user to fix something the system broke — it is the closest available proxy for the defect the platform exists to eliminate, and it was the number reported to the executive sponsor.

18 Runbooks

Organized by failure mode rather than by component, because at three in the morning nobody knows which component it is.

Brokerage outage — circuit open, connections accumulating

1Confirm the circuit is open for one brokerage only. If several opened together, suspect the platform's own egress or DNS before suspecting the vendors.

2Check whether the vendor has published an incident. Record the reference in the incident channel — reconciliation of what the vendor said against what was observed matters at the post-incident review.

3Verify that other brokerages show normal queue depth and latency. If they do not, isolation has failed and that is the larger incident.

4Confirm affected connections moved to Broker Outage rather than Reauth Required. If any reached Reauth Required, the error map is wrong and users are being asked to fix a vendor problem — stop and correct the mapping first.

5Assess expiry risk. If credentials will expire before the vendor plausibly recovers, escalate to product for user communication rather than waiting for the failure to arrive on its own.

6Do not raise retry budgets to push through. That converts a vendor incident into an allowlist or reputation problem, and it does not work.

Dead-letter queue growth

1Group by brokerage, error class and correlation identifier before touching anything. Replaying an undiagnosed dead-letter queue reproduces the original failure at speed.

2Confirm every affected connection's current version and fence. Anything whose version has advanced is stale and must be discarded, not replayed.

3For rotating brokerages, run adapter validation before replaying anything. Replaying a superseded credential is the one action that turns a queue problem into a user problem.

4Replay in a bounded batch and confirm the commit rate before releasing the rest.

5Any message that dead-letters twice becomes a defect ticket, not a third replay.

Reauthorization spike

1Split platform-originated from brokerage-originated immediately. These have entirely different responses and conflating them wastes the first hour.

2If platform-originated: check for a recent adapter release, an error-map change, or unknown vendor codes appearing in telemetry. An unmapped code defaulting incorrectly is the most common cause.

3If concentrated on one brokerage: compare against the capability matrix. A vendor changing rotation or lifetime behaviour without notice presents exactly this way.

4If concentrated on one region: check ownership churn and fence increment rate. Ownership thrashing produces reauthorization as a downstream symptom.

5Suspend the affected cohort before notifying users. A user who re-authorizes into a still-broken path re-authorizes twice and stops trusting the prompt.

Suspected multiple-owner anomaly

1Treat as a correctness incident from the first minute. This condition should be structurally impossible, so its appearance means an assumption is wrong.

2Suspend the affected connections immediately. Availability is the cheaper loss here.

3Capture the fence sequence, the ownership records and the commit history before anything is remediated. This evidence is not reconstructable afterwards.

4Verify the global authority's own health. A conditional write silently degrading is the only plausible route to this state.

5Do not resume until the mechanism is understood. A connection that refreshes successfully after an unexplained multiple-owner event has not been shown to be safe.

Regional failover

1Confirm the failure is regional rather than a dependency shared across regions. Failing over from a shared-dependency failure moves the outage without ending it.

2Verify the global authority is reachable from the surviving region. If it is not, that region is correctly failing closed and failover will not proceed — this is the design working, not a fault.

3Confirm old leases have expired or been revoked before granting higher fences.

4Transfer ownership in brokerage-ordered batches, watching commit success and brokerage rate limits. A whole estate re-owning at once looks like an attack to a vendor.

5Monitor expiry risk during the transfer — the recovery objective is about ownership, and the deadline that matters is credential expiry.

Failback to a recovered region

1Never automatic. Failback is a deliberate, scheduled transfer with a named owner.

2Confirm the recovered region holds no stale in-flight work. Anything it holds predates the failover and is invalid by definition.

3Verify it reads current committed state from the authoritative store, not from any cached or replicated view it retained.

4Transfer with a fence increment, in batches, verifying that no commit carries an older token version.

5Hold a full expiry cycle before returning the region to normal share. There is no operational reason to hurry this, and one very good reason not to.

Suspected credential disclosure

1Revoke first, investigate second. A credential that may have leaked is worth less than the time spent establishing whether it did.

2Identify the exposure path — telemetry, a support surface, an exception report, a dependency — and establish scope by connection, brokerage and time window.

3Preserve audit evidence before remediating. Retention windows are finite and the investigation will outlast some of them.

4Check whether canary detection fired. If it should have and did not, the detection gap is a second incident that needs its own record.

5Engage compliance and the brokerage relationship owner on notification obligations. This is a contractual and regulatory question, not an engineering one.

6Close the serialization gap that permitted it before restoring normal operation. A redaction gap that is understood but open will recur.

Key management unavailable

1Expect refreshes to fail closed. Workers cannot decrypt and must not proceed — this is correct behaviour, and the incident is availability, not correctness.

2Confirm scope by region and by key. A single replica's policy or grant problem looks identical to a service impairment from inside one region.

3If a recent grant or policy change preceded it, suspect that first. Authorization changes can take time to become consistent, and the mitigation for that window is a grant token rather than a retry loop.

4Do not move ownership to another region hoping it can decrypt. If a replica and its grants were not deliberately established there, it cannot — and the attempt adds ownership churn to an existing incident.

5Track expiry risk throughout. This failure mode is silent until credentials start expiring, and then it is not.

19 Decision record

Recorded with reasoning intact, so the next architect inherits the decision rather than guessing at it.

ADR	Decision	Reasoning	Owner & revisit
ADR-01	Connection is the ownership boundary	Users hold multiple logins; logins expose multiple accounts. Scheduling, locking, ownership, versioning and recovery all attach here or the model is wrong.	Architecture · stable
ADR-02	Active-active platform, active-passive per connection	Independent work parallelizes safely; one rotating credential does not. The two operate at different granularities and do not conflict.	Architecture & Platform · stable
ADR-03	Transactional metadata and ciphertext as source of truth; caches non-authoritative	A cache that can be authoritative will eventually be authoritative and wrong, at the worst moment.	Architecture · stable
ADR-04	Envelope-encrypted database fields for user credentials; managed secrets service for platform credentials	Different volumes, access patterns and rotation models. Conflating them costs on both dimensions.	Security · annual
ADR-05	Separate stores for the commit path and the ownership authority	Multi-row transaction versus cross-region strongly consistent compare-and-set. The strongly consistent mode that gives the fence its guarantee also forbids transactions and fixes the region set — constraints a single-item ownership record absorbs easily and a multi-row commit path cannot. Splitting confines those constraints to the store that tolerates them rather than imposing them on the commit.	Architecture · 18 months
ADR-06	Durable expiry-driven schedules plus independent reconciliation	In-memory polling loses work on deploy. Reconciliation is the working assumption that scheduling will fail.	Architecture & Platform · stable
ADR-07	No exactly-once claim	At-least-once delivery with idempotent handlers and an atomic conditional commit. Queue deduplication is bounded by a finite window and does not survive a consumer that acts then dies. Correctness rests on the commit.	Architecture · stable
ADR-08	Fail closed during a control-plane partition	Refreshes are lost during the partition; a wrongly committed rotation is not recoverable. Approved with the availability cost stated.	CTO & Architecture · stable
ADR-09	Per-brokerage limits, breakers and queue partitions	One vendor's throttling must not starve the others. Isolation designed in, not added after the first incident.	Platform · per integration
ADR-10	Single-region multi-AZ first, multi-region after correctness proven	Adding geography to an unproven concurrency model multiplies the failure surface without addressing the failure actually occurring.	CTO & Product · closed
ADR-11	Token platform never triggers order retry	Coupling credential recovery to order execution turns a token incident into a duplicate-fill incident. Ambiguous responses are reconciled under an idempotency key.	Trading & Risk · stable
ADR-12	AI confined to read-only telemetry analysis	Anomaly detection earns its place. Autonomous action on credentials offers small upside against a failure mode only the end user can repair.	Security & Architecture · annual
ADR-13	Three concurrency layers retained — bounded lease, version compare-and-swap, monotonic fence	Each covers a failure the other two do not: a lease is defeated by a paused worker, a version check is defeated by a recently-read former owner, and the fence is what makes recovery safe. The fence is enforced in the commit store's high-water mark, not only in the issuing authority.	Architecture · stable
ADR-14	Service objectives published as design constraints, with measurement definitions fixed before the build	An objective derived after the fact is a description. Fixing the denominator, the exclusions and the fail-closed position in advance is what makes the number mean anything later.	Architecture & Product · annual

20 Migration interlock

Migration is where credentials are actually lost. Steady-state operation is comparatively safe.

The one unrecoverable migration failure is two systems rotating the same credential. Any window in which both the legacy refresher and the new platform can act on one connection will eventually be entered — by a delayed deployment, a cached configuration, a rollback, or a queue that was drained more slowly than assumed. Making that state unreachable is worth more than any quantity of care about not entering it.

Atomic transfer. Ownership of a connection moves in a single committed step. There is no intermediate state in which both systems consider themselves responsible.
Technical interlock, not configuration. After transfer, the legacy path is prevented from acting — the attempt fails and is recorded. A feature flag that could be flipped back by a rollback is not an interlock.
Cohorts by brokerage and risk. Sequenced so the highest-risk rotating integrations migrate once the pattern is proven on a forgiving one, not first and not last.
A full expiry cycle per cohort. No cohort promotes until every connection in it has refreshed at least once under the new platform. A migration that has not survived a refresh has not been tested.
A rollback path per cohort. Ownership returns cleanly, with the same atomicity guarantee in the reverse direction.
Decommission only after the last cohort holds. The legacy refresher is removed, not merely disabled, so it cannot be revived by a deployment that predates the migration.

21 References

Standards and primary sources this design was built against. Vendor and brokerage documentation used during the engagement is not listed, since naming it would identify the integrations.

RFC 9700 — Best Current Practice for OAuth 2.0 Security (BCP 240, January 2025). §4.14.2 on refresh-token protection and replay detection; §2.2.2 on sender-constrained tokens and rotation.
RFC 6749 — The OAuth 2.0 Authorization Framework (October 2012). §6 on refreshing an access token.
RFC 7636 — Proof Key for Code Exchange (PKCE).
RFC 9449 — OAuth 2.0 Demonstrating Proof of Possession (DPoP). Application-level sender constraining, usable by public clients.
RFC 8705 — OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens.
draft-ietf-oauth-v2-1 — The OAuth 2.1 Authorization Framework. An active Internet-Draft, not a standard; cited as direction of travel only.
Kleppmann, "How to do distributed locking" (2016). The canonical statement of why a lease without a fencing token does not hold under process pauses.
Amazon DynamoDB global tables — consistency modes. The distinction between conditional writes evaluated against the local Region's copy and those evaluated against the latest version globally is the crux of the ownership-store decision.
AWS KMS multi-Region keys. "Multi-Region keys are not global" — replicas are deliberate, and each carries its own policy and grants.
Amazon SQS FIFO — exactly-once processing. Deduplication is bounded by a finite interval, which is why correctness rests on the conditional commit rather than on delivery semantics.
Amazon EventBridge Scheduler. At-least-once delivery and one-time schedules. Automatic removal of completed one-time schedules is a separate configured behaviour — see deleting a schedule and the ActionAfterCompletion parameter.
Aurora global database — disaster recovery. Recovery characteristics for planned switchover versus unplanned failover.
AWS Fault Injection Service — actions reference. Connectivity disruption scoped to an availability zone, and cross-region route-table disruption, behind T-10, T-13 and T-15. Regional isolation, not region destruction — no service simulates the latter, and this reference does not claim it did.
OpenTelemetry documentation. Trace and metric instrumentation, with redaction applied at the serialization layer.