Multi-Region Brokerage Connectivity and Token Resilience Platform

Executive overview

Architecture and delivery of a dedicated token-resilience control plane for an automated trading platform — one safe refresh owner per brokerage connection across multiple regions, with leases, fencing and versioned atomic commits replacing an in-process refresher that lost credentials under concurrency.

Assessment artifacts

The working reference from this engagement, in a self-contained interactive page: the component architecture, the connection state machine, the fenced refresh sequence with its crash points, the brokerage capability matrix, the multi-region ownership and failover model, the service-level objectives, the threat model, the concurrency and failure test matrix, and runbooks organized by failure mode.

Business challenge

The client's platform submits orders automatically on behalf of its users, and every one of those orders depends on a live brokerage authorization. Those authorizations are short-lived. Access tokens expire in minutes at some brokerages, refresh tokens expire in days at others, and at several the refresh token itself rotates — the old value is invalidated the moment a new one is issued. The refresh logic lived inside the trading application as a periodic in-process job. That arrangement had three failure modes the business had already met. When the application was redeployed or degraded, refreshes stopped and connections silently aged out. When two application instances refreshed the same connection at once, one of them presented a value the brokerage had already invalidated, and the user's whole authorization was revoked — not degraded, revoked, requiring the user to log in to their brokerage again. And when a worker died after the brokerage had responded but before the new token reached the database, the platform's record of the credential and the brokerage's record diverged with no mechanism to detect it. The visible symptom was always the same and always arrived late: a user's automated strategy stopped trading, and nobody knew until the user noticed. The real problem was not reliability engineering. It was that the system had no concept of who owned a credential at a given moment, so it had no way to prevent two parties from acting on it.

Environment and constraints

• A privately held United States wealth-technology company operating an automated trading platform; client identity, brokerage relationships, user counts, volumes and commercial terms are withheld.
• Users hold connections to more than one brokerage, and may hold several separately authorized logins at the same brokerage, each exposing one or many tradable accounts.
• Brokerage authorization behaviour is heterogeneous and not negotiable. Rotation, lifetimes, session exclusivity, keepalive requirements and error semantics differ per vendor, and the platform has no influence over any of them.
• Orders are submitted automatically, so a lapsed credential has direct financial consequence for the user and is not merely a degraded experience.
• A SOC 2 examination was in progress, so control design and evidence retention were in scope from the start rather than retrofitted.
• A small senior engineering team, with no dedicated site-reliability function at kickoff; an on-call platform rotation was established during delivery and owns the operational decisions recorded in the decision register.
• The legacy in-process refresher could not be switched off in one step; migration had to be staged, reversible, and incapable of letting both systems rotate the same credential.
• Some brokerages require connections to originate from allowlisted egress addresses, which constrains network topology and complicates regional expansion.
• There is no maintenance window that suspends the obligation to hold sessions. Expiry is driven by the brokerage clock, not the client's release calendar.

Objectives and success measures

• Maintain brokerage authorization independently of the availability of the primary trading application.
• Guarantee one safe refresh owner per brokerage connection while the platform as a whole operates across multiple regions.
• Support multiple brokerages, multiple logins per user, and one or many accounts beneath each connection, with accurate normalized status for each.
• Prevent credential loss caused by concurrent refresh, rotating refresh tokens, stale workers, network partitions or incomplete commits.
• Integrate token readiness with automated trading without creating blind order retries.
• Make brokerage onboarding a certified adapter exercise governed by a versioned contract rather than a bespoke integration.
• Produce the control design and retained evidence required for SOC 2 examination.
• Leave a platform the client's own engineers can operate, extend and reason about without the architect.

Role and responsibilities

Principal architect and hands-on delivery lead. I owned the target architecture, the consistency model and the architecture decision records, and implemented the reference paths the rest of the platform was built against — the token state machine, the lease and fencing path, and the reference broker adapter. I defined the adapter contract and its certification suite, authored the threat and load models and the failure-injection plan, and led a small senior team through build, staged migration and disaster-recovery validation. I presented the production readiness review to the executive sponsor, who held the go decision.

Architecture and design approach

• I started by establishing what each brokerage actually does, because the entire design depends on it and none of it can be assumed. The capability matrix recorded, per integration: authorization type, whether the refresh token rotates on use, access and refresh lifetimes, whether a session keepalive is required independently of token refresh, whether a session is exclusive to one active client, whether credentials are scoped to the login or to individual accounts, rate limits, fixed-egress-IP requirements, and observed recovery behaviour after an error. Vendor documentation gave the happy path. The columns that mattered most — what happens on a concurrent refresh, what a particular error code really means, whether a rotation had taken effect before the connection dropped — came from controlled testing in sandbox and from the platform's own production failure history.
• That matrix drove the single most consequential decision in the project: the unit of ownership. The obvious candidates were the user and the account, and both are wrong. A user may hold several separately authorized logins at the same brokerage. One login may expose many tradable accounts that share a credential. Refresh, locking, regional ownership, token versioning and recovery all attach to the separately authorized connection, and nothing else. Accounts inherit the connection's credential unless a brokerage explicitly issues account-scoped material. Getting this boundary wrong produces a system that looks correct in testing and corrupts connections in production the first time a user adds a second login.
• I then wrote the invariant the platform exists to hold, in one sentence, and made everything else answer to it: at most one current owner may commit a refresh result for a given connection and token version, and a stale worker or region may execute a refresh but must not be able to commit once a higher fence exists. Stating it that way made the design tractable. It also made it testable, which mattered more.
• Concurrency control is three layers, not one, because each catches what the others miss. A bounded worker lease keeps two workers in the same region from doing redundant work — but a lease is only a hint, since a worker can pause past its expiry and wake up believing it still holds the lock. An optimistic compare-and-swap on the connection's token version makes the commit itself conditional, so a worker operating on a stale read cannot overwrite newer state. A monotonically increasing fencing token, issued by a strongly consistent global authority and required on every state-changing commit, is what makes a recovered former owner harmless: its fence is lower than the current one, and the commit is rejected at the database rather than at the application's discretion. A distributed lock without a fence is a suggestion, and the platform's earlier incidents were exactly what that looks like.
• Scheduling is durable and expiry-driven rather than periodic. Each connection carries a persisted next-refresh time derived from its actual expiry, less a per-brokerage safety buffer, plus jitter to keep a fleet of connections from converging on the same second. An independent reconciliation scanner runs on a short cycle and looks for what scheduling alone cannot fix: connections overdue, connections with no schedule at all, connections stuck mid-refresh, and connections whose state contradicts their token. Scheduling is the mechanism; reconciliation is the reason the mechanism is allowed to fail. In-memory timers were excluded by design — they are the reason the previous system's refresh work vanished on deploy.
• The commit is a single transaction. New token ciphertext, expiration, incremented version, next scheduled refresh and connection state land together or not at all, conditioned on both the expected version and the current fence. A partial commit here is indistinguishable from corruption, because the platform would hold a credential it could not prove was current.
• The hardest case in the whole system is a worker that dies after the brokerage has responded and before the local commit — at a rotating brokerage, the old refresh token is already dead and the new one exists only in the lost process. I handled it by making the attempt itself durable. A worker records its intent — connection, expected version, fence, correlation identifier — and commits that record before it contacts the brokerage. That does not recover the lost credential; nothing can. What it does is convert a silent divergence into a known-ambiguous state, which is the difference between a problem the platform can act on and one it discovers when a user complains. Reconciliation finds the stranded attempt and does not retry blindly with the old value, because at a rotating brokerage that is precisely the action that triggers replay detection. It establishes live state through the adapter first — and the design is honest that a plain validation call is a weak signal here, since a successful refresh does not invalidate the access token that preceded it, so discriminating a rotation that took effect from one that did not requires a capability the adapter has to declare it has. Where no such signal exists, the connection is declared reauthorization-required rather than retried into a revocation.
• Brokerage integrations sit behind a versioned adapter contract rather than in the core. Each adapter declares its capabilities, implements authorize, refresh, validate and revoke, and maps its own error vocabulary onto a normalized set the platform reasons about — retryable, rate-limited, brokerage-unavailable, terminal, and user-action-required. That last distinction is worth the whole framework: treating every rejection as user-action-required trains users to re-authorize when nothing was wrong, and treating every rejection as retryable hammers a brokerage that is already unwell. Every adapter passes a certification suite before it reaches production, and adapters are versioned so a brokerage's behaviour change is a controlled rollout rather than an incident.
• Isolation was designed in rather than added after the first outage. Each brokerage gets its own queue partition, rate limiter and circuit breaker, so a slow or throttling brokerage consumes its own capacity and nobody else's. Without that, one brokerage's rate-limit response backs up a shared queue and starves every other integration — the platform then fails in aggregate for a fault that was confined to one vendor.
• The boundary with trading was deliberately narrow and is the control I would defend first. The token platform exposes a usable-token contract and a readiness signal; it never calls the trading path, never triggers an order retry, and trading services have no access to the credential store. An order that received an ambiguous response is reconciled against the brokerage before anything is resubmitted, under an idempotency key. Coupling credential recovery to order retry is how a token incident becomes a duplicate-fill incident, and a duplicate fill is a materially worse outcome than a missed one.
• The design takes an explicit position that during a control-plane partition, a region without current authority fails closed for refresh commits rather than proceeding on a stale belief about ownership. That is a choice of safety over availability, it costs refreshes during the partition, and some of those become user-visible reauthorizations. It was put to the executive sponsor with that cost stated rather than absorbed as an implementation detail, because a rule decided during an incident gets reversed during the incident, and the reversal is what causes the damage.

Security and governance considerations

• Credential material is encrypted in transit under TLS and at rest under KMS envelope encryption, with separate key boundaries per environment and per region. Platform application credentials sit in a managed secrets service; high-volume user token material sits in encrypted database fields, because the access pattern and the rotation model are different and conflating them costs both.
• Decryption is reachable only by the token service's own workload identity and by audited break-glass roles. No human path to plaintext exists in the ordinary course, and support tooling was built to operate on identifiers and normalized status rather than on credentials — a support user cannot see a token because the interface has no way to render one.
• No credential ever leaves the service boundary in a payload. Scheduled jobs, queue messages, events, logs, traces, metrics, analytics extracts and exception reports carry connection identifiers, versions, fences and correlation identifiers only. This is enforced at the serialization layer rather than left to reviewer discipline, because redaction that depends on every future contributor remembering is not a control.
• Regional decryption is constrained deliberately. Replicating a multi-Region key does not make it global — a replica exists only where one was created, and each replica carries its own key policy and grants, so a region's ability to decrypt is an explicit authorization decision. Replicated ciphertext does not confer the right to use it.
• The authorization callback is treated as attacker-reachable, because it is: state validation, PKCE where the brokerage supports it, single-use callback state, a strict redirect allowlist and replay protection.
• Service-to-service calls are authenticated and authorized on workload identity. Trading services hold no credentials for the token database and reach the platform only through its API — the strongest control in the design is an absent connection string.
• Credential reads, refreshes, revocations, reauthorizations, ownership transfers and privileged actions emit tamper-evident audit events. Retention and secure deletion of revoked credentials and operational history follow the retention position agreed with the client's compliance function rather than a default.
• Leakage is monitored for rather than assumed away. Synthetic canary credentials are planted where a leak would surface, and logs and traces are scanned automatically for credential-shaped material. A control that is never tested is a belief.
• Artificial intelligence is deliberately confined to read-only work. Anomaly detection over refresh, ownership and brokerage-health telemetry surfaces conditions a static threshold misses — a brokerage degrading slowly, a single connection retrying abnormally, a regional pattern shifting before it breaches. No automated agent may author a token state transition, acquire a lease, decrypt credential material or initiate an ownership transfer. AI-assisted development was permitted with human review, and excluded from the concurrency and cryptographic paths, where a plausible-looking change is the most dangerous kind. This boundary is written into the governance record rather than left as a convention, because conventions erode.
• The threat model was written against named scenarios and revisited at each gate: two workers presenting the same rotating credential; a crash between brokerage success and local commit; a stale region returning after another obtained a higher fence; a database failover during rotation; malformed or incomplete brokerage responses; an unrelated service or support user attempting to read plaintext; a brokerage outage causing queue growth and cross-brokerage starvation; and a replayed or redirected authorization callback.

Implementation and migration approach

• Discovery before design. Current-state inventory, the brokerage capability matrix built from testing rather than documentation, a dependency map, an honest reading of the platform's own failure history, a first threat model and the list of decisions that would need architecture decision records. Reviewed with product, trading, platform, security and operations together, because a token platform touches all five and a decision agreed with only one of them does not survive.
• Target architecture and consistency model. Data model, connection state machine, API and adapter contracts, the argument for why the lease, version and fence combination holds the invariant, the service-level objectives and the capacity model. This produced an architecture review sign-off and an implementation backlog rather than a document.
• A production-grade vertical slice for one pilot brokerage, not a prototype. Connection registration, encrypted storage, durable scheduling, the refresh worker, reconciliation, the usable-token API and its dashboards — built to the standard the platform would ship at, so that what followed was replication rather than rewriting.
• Multi-availability-zone hardening, brokerage isolation, dead-letter handling and replay, the reauthorization flow, the first migration cohort and the runbooks. The gate here was a game day: availability-zone loss and deliberate worker termination, with the specific requirement that the legacy and new refresh paths be demonstrably incapable of acting on the same connection.
• A second adapter, the multi-region proof, the global fencing test and the production readiness review. A controlled regional isolation exercise using fault injection, not a tabletop — connectivity severed rather than a region pretended out of existence, which is the honest limit of what can be tested — including the recovery of the isolated region and its failback, which is the direction that actually loses credentials, because a returning owner holds state it believes is current and is not.
• Migration in cohorts with a hard interlock. Ownership of a connection transfers to the new platform in a single atomic step, after which the legacy refresher is technically prevented from acting on it. Cohorts were sequenced by brokerage and by risk, each with a rollback path, and no cohort proceeded until the previous one had held through a full expiry cycle. Two systems rotating the same credential is the one migration failure with no recovery, so the design made it unreachable rather than unlikely.
• Storage was split along the grain of what each part needs. The commit path stays in a transactional relational store, because token ciphertext, expiry, version, schedule and state must move together in one transaction and that is what a relational database is for. Global ownership and the fence live in a separate strongly consistent store with conditional writes, because that decision has to be correct across regions and is a single-item compare-and-set that needs no transaction. Both stores support conditional writes evaluated against current state; neither is asked to do the other's job.
• Observability was built as part of the platform rather than added for the readiness review. Expiry risk with a viable-path indicator; queue depth, oldest-message age, redelivery and dead-letter counts by brokerage and region; refresh success, latency, attempts and completion-before-expiry; lease failures, fence increments, regional moves and any multiple-owner anomaly; normalized brokerage health; and the security signals — denied decrypts, unauthorized service calls, leak indicators and break-glass use. Every dimension is redacted by design.
• Testing was the deliverable, not the check on it. Deterministic concurrency tests around the rotation race; worker termination at each defined crash point; database failover during rotation; queue duplication and redelivery; clock skew; regional loss; and adapter contract certification per brokerage. Failure injection ran against real infrastructure rather than mocks, because the failures that matter are the ones the mocks do not model.
• Handover as a work item with its own acceptance. Architecture decision records with their reasoning intact, adapter authoring documentation, runbooks organized by failure mode rather than by component, dashboards, a recurring game-day schedule, and working sessions with the engineers who would own it. The client engaged an architect, not a dependency.

Key decisions and trade-offs

• The separately authorized brokerage connection is the boundary for scheduling, locking, ownership, versioning and recovery — not the user and not the account. Nearly every correctness argument in the platform reduces to this choice, and the two intuitive alternatives are both wrong in ways that only appear in production.
• Active-active at the platform level, active-passive per connection. Different users and different connections run in any healthy region simultaneously; a single connection has exactly one current owner region. Active-active refresh of one rotating credential is not a performance optimization, it is the original defect.
• Fail closed during a control-plane partition. A region without current authority stops committing refreshes rather than proceeding on a stale belief. This costs availability in a rare case to protect correctness in a case that cannot be undone, and it was approved explicitly at executive level rather than buried in a design document.
• Three concurrency layers, not one. A lease alone is defeated by a paused worker; a version check alone does not stop a former owner region that read recently; the fence is what makes recovery safe. Reviewers consistently proposed removing one of the three, and the reason to keep all three is that each covers a failure the others do not. The fence is additionally recorded as a high-water mark in the store that accepts the commit, not only in the authority that issues it — otherwise it is checked nowhere that matters, and the interval between a lease expiring and a new owner's first write is unguarded.
• The commit path and the ownership authority live in different stores, chosen for different properties. The strongly consistent cross-region mode that gives the fence its guarantee also forbids multi-row transactions and fixes the set of regions in advance — constraints a single-item ownership record absorbs without difficulty and a multi-row credential commit cannot. Splitting confines those constraints to the store that tolerates them instead of imposing them on the commit path. The reasoning, and the conditions under which it should be revisited, are recorded in an architecture decision record with a review date, because this is the decision most likely to be overtaken by what the platforms ship next.
• Durable expiry-driven schedules plus independent reconciliation, never in-memory polling. Reconciliation is not redundancy; it is the assumption that scheduling will fail, made operational.
• No exactly-once claim. The platform provides at-least-once execution with idempotent handlers and an atomic conditional commit, and says so. Queue-level deduplication is bounded by a finite window and does not survive a consumer that acts and then dies, so correctness rests on the conditional commit rather than on delivery semantics. A design that claims exactly-once network execution has usually just moved the ambiguity somewhere nobody is looking.
• Single-region multi-availability-zone first, multi-region only after correctness was proven. Adding geography to a design whose concurrency model has not been demonstrated multiplies the failure surface without addressing the failure that was actually occurring.
• The token platform never triggers an order retry. Separating credential recovery from order execution is the control that keeps a token incident from becoming a duplicate-fill incident, and it constrains both systems permanently by design.
• Artificial intelligence stays out of the commit path. Anomaly detection over telemetry earns its place; automated action on credentials does not. The value of an agent acting autonomously here is small and the failure mode is a revoked authorization the user must personally repair.
• Service objectives were written as constraints the design had to satisfy, with their denominators and exclusions fixed before anyone could see the results. An objective defined after the numbers are in is a description of what happened, and it cannot be failed.

Results and outcomes

• Brokerage authorization survives the trading application. Token refresh runs in a separate service with its own availability envelope, so a deployment, a degraded release or an application incident no longer ages out user connections as a side effect.
• The class of failure that caused the original incidents is now structurally prevented rather than made less likely. A stale worker or a recovered former owner region cannot commit over newer state, because the commit is conditioned on both the token version and a monotonic fence and is rejected by the database rather than by application courtesy.
• Multi-region failover and failback were exercised under controlled regional isolation — connectivity severed by fault injection rather than a region simulated as destroyed, which no service can do — and produced no stale commit. Failback is a deliberate ownership transfer with a fence increment, not an automatic return, which is what stops a recovered region from copying older credential state over a newer rotation.
• The behaviour of every integrated brokerage is written down and maintained rather than held in the memory of whoever built the integration. The capability matrix records rotation behaviour, lifetimes, session and keepalive semantics, scoping, limits and recovery, and it is the artifact a new integration starts from.
• Adding a brokerage is a contract exercise instead of a project. A new adapter implements a defined interface, declares its capabilities, maps its errors onto the normalized set and passes a certification suite before it can reach production.
• One brokerage's outage or throttling no longer degrades the others. Per-brokerage queue partitions, rate limiters and circuit breakers confine the fault to the integration that caused it.
• Reauthorization became a designed product experience rather than a support outcome. The platform distinguishes a brokerage that is temporarily unwell from a credential that genuinely requires the user to act, and only the second reaches the user.
• Blind order resubmission was eliminated as a class. The token platform never triggers a retry on the trading path, and an ambiguous order response is reconciled against the brokerage under an idempotency key before anything is resubmitted.
• The legacy in-process refresher is technically prevented from acting on migrated connections rather than switched off by configuration, which is what made a staged, reversible migration possible without two systems ever rotating the same credential.
• The control environment produces evidence rather than assurances. Credential access, refresh, revocation, ownership transfer and break-glass use emit tamper-evident audit events, and the platform's own telemetry is scanned for credential material with synthetic canary credentials as the detection tripwire.
• The availability and refresh-completion objectives were set at architecture review as design constraints and measured continuously against a published definition, with the denominators, exclusions and the fail-closed position agreed before the build rather than reconstructed afterwards. Results are held with the client and are not published here.
• The team operates and extends the platform without depending on any one person. Architecture decision records, the adapter contract, runbooks per failure mode, dashboards and a recurring game-day schedule were delivered as part of the work rather than after it.

Lessons learned

• A distributed lock without a fencing token is a suggestion. Any lock holder can be paused — by garbage collection, by a hypervisor, by a network partition — past its own expiry and wake up believing it still holds the lock. The only defence that survives that is a monotonically increasing token checked at the point of commit, in the store, where the application's belief about itself is irrelevant.
• The window between a successful brokerage response and the local commit is the entire problem, and most designs skip it. Everything else in a token platform is comparatively ordinary engineering. If a candidate architecture has no answer for that window, it has no answer.
• Never retry blindly into a brokerage that rotates credentials. Presenting a value that has already been superseded is exactly what replay detection is built to catch, and the response is typically to revoke the whole family — converting a recoverable local failure into one only the user can repair. The recovery path has to establish live state before it acts.
• Vendor documentation describes the happy path. Concurrency behaviour, real error semantics and recovery are learned by testing, and they change without notice. The capability matrix decays, and it matters most at the moment it is wrong — which is why it is a maintained artifact with an owner rather than a discovery deliverable.
• Reauthorization is a product problem wearing an infrastructure costume. Once the platform could distinguish a brokerage that was temporarily unwell from a credential that genuinely required the user to act, the volume reaching users fell on its own — not because refresh got better, but because the system stopped asking for help it did not need.
• Migration is where credentials are actually lost, not steady-state operation. Any window in which two systems can both rotate one credential will eventually be entered. Making that state unreachable is worth more than any amount of care about not entering it.
• Fail-closed has to be decided before the partition, in daylight, with the people who own the availability number in the room. Decided during an incident, it will be reversed, and the reversal will be the thing that causes the damage.
• Isolation between third-party integrations is not a scaling concern to defer. One vendor's rate limiting fills a shared queue and starves every other integration, and the platform then fails in aggregate for a fault that belonged to one vendor.
• Correctness arguments have to be legible to people who will never read the code. The invariant written in one sentence did more to hold the design together across a year of delivery than the proof behind it, because it was the thing everyone could remember and check a proposal against.

Related technologies

• OAuth 2.0 authorization code flow
• PKCE (RFC 7636)
• Refresh token rotation and replay detection
• OAuth 2.0 Security Best Current Practice (RFC 9700)
• Amazon Aurora PostgreSQL
• Amazon DynamoDB global tables
• Amazon EventBridge Scheduler
• Amazon SQS (FIFO and standard) with dead-letter queues
• AWS KMS envelope encryption
• AWS KMS multi-Region keys
• AWS Secrets Manager
• Amazon ECS on AWS Fargate
• AWS Fault Injection Service
• NAT gateway Elastic IPs for brokerage allowlisting
• Terraform
• Kotlin and Java on the JVM
• Go
• Optimistic concurrency and conditional writes
• Bounded leases and monotonic fencing tokens
• Transactional outbox and durable intent records
• Idempotency keys
• Circuit breakers, bulkheads and backpressure
• OpenTelemetry
• Amazon CloudWatch anomaly detection
• Testcontainers and deterministic concurrency testing
• SOC 2 Trust Services Criteria

Related projects