Operations & SRE
SLO mechanics, observability standards, incident and problem management, capacity, backup/recovery operations, AIOps stance, and telemetry cost/privacy controls. Companion views: Observability & SRE and Resilience & DR; procedures live in the runbook library.
Honesty header: every numeric target below is a placeholder band until the BIA (P1) and volume data (OQ-04) exist — the framework is real, the numbers are not yet (A-05, A-10).
1 · SLIs, SLOs & error budgets
Every service in the catalog declares its SLIs from a standard menu — availability (good requests / total), latency (p95/p99 against its NFR band), quality (error class rates), freshness (data products), lag (event consumers) — and sets SLOs per its service tier at onboarding (RB-05). The error budget is the operational contract: while budget remains, teams ship; when a budget burns down within its window, the published error-budget policy activates — feature work yields capacity to reliability work, risky deploys pause, and the reliability review (gov-reliability) tracks the recovery. Budgets make the reliability/velocity trade explicit instead of political (S37 formalized; thresholds placeholder until tiers are BIA-confirmed).
| Tier | SLO stance (placeholder bands — BIA binds numbers) | Review cadence |
|---|---|---|
| T1 mission-critical | Tightest availability/latency SLOs; multi-AZ mandatory; synthetic coverage mandatory | Monthly reliability review + per-incident |
| T2 business-critical | Strong SLOs; multi-AZ mandatory; synthetics on key journeys | Monthly |
| T3 important | Standard SLOs; single-region multi-AZ preferred | Quarterly |
| T4 supporting | Best-effort with defined support hours; vendor SLAs where SaaS | Quarterly |
2 · Telemetry standards
OpenTelemetry is the instrumentation standard (ADR-11): SDKs with enterprise semantic conventions (service.name from the catalog, environment, tier, owner labels), collectors per cluster/zone performing sampling and redaction, one consolidated backend for metrics, logs and traces. Correlation ids propagate end-to-end — channel → gateway → BFF → services → events (trace context in headers and event metadata) → data pipelines — so a customer complaint can be walked to the exact failing hop. Service maps derive from real trace topology, not diagrams; the canonical model and the observed map are reconciled quarterly (drift between them is a finding). Golden-path templates ship instrumentation pre-wired, which is why coverage is an adoption metric rather than a campaign.
3 · Alert ownership & routing
Three rules: every alert has a named owning team (orphan alerts are deleted or adopted at the monthly audit); alerts page on symptoms and SLO burn, not raw causes (cause-level signals become dashboard context); and every page links a runbook or it doesn't page. Routing follows catalog ownership through the paging service (ops-incident) with tiered urgency: page (now), ticket (business hours), FYI (digest). Noise is managed as debt — the alert-quality review tracks pages-per-shift and actionability rate, and noisy rules lose their paging rights until fixed.
4 · Incident & problem management
Incidents run per RB-23: severity matrix, incident commander at Sev-2+ (command separate from keyboard), stabilize-first doctrine (rollback / failover / degrade / scale before root-causing), honest comms on cadence, and verification via synthetics before closure. Security suspicion pulls in RB-16 immediately — availability restoration must not destroy evidence. Problem management is the follow-through: blameless post-incident reviews within days placeholder produce actions with owners and dates; repeat incidents from unactioned PIR items are a reliability-review escalation, because a lesson not implemented is a lesson not learned. Trends (by service, by cause class) feed the risk register and the roadmap's optimization backlog.
5 · On-call & escalation
Teams own the pager for what they build (domain services, data products, platform components alike). Rotas are staffed to a humane density placeholder; escalation ladders are short and named: primary → secondary → team lead → duty manager → duty executive (crisis). Platform and security maintain their own 24×7-capable rotas placeholder pending org sizing. Every on-call joiner completes a readiness checklist (access verified via PAM, runbooks read, shadow shift done). Pager health is reviewed monthly: page volume, sleep-hours pages, handoff quality — sustained overload is treated as a capacity or architecture defect, not a personal one.
6 · Change, release & configuration
Standard changes flow through golden-path pipelines with GitOps delivery (RB-13) — the pipeline evidence is the change record for low-risk classes, keeping the ITSM overhead proportional to risk. Higher-risk changes add explicit approval and verification plans; freeze windows protect cutovers and peak periods with RB-14 as the only exception path. Configuration truth lives in Git (IaC + manifests) reconciled by controllers; the CMDB holds service-level records (ownership, tier, dependencies) synchronized from the catalog and this package's model — drift between deployed reality, Git and the model is detected (C-CLD-02) and treated as an incident, which is what keeps the architecture repository honest over time.
7 · Capacity management
Capacity runs on telemetry, not folklore (RB-18): each component's capacity record (in the model) names its scaling triggers, owner and test method; dashboards project utilization against growth assumptions; reviews happen on cadence and before known load events, with pre-scale/post-scale-down discipline enforced by FinOps checks. Autoscaling is bounded by quotas so a bug cannot buy infinite compute. Until OQ-04 volumes exist, all thresholds are placeholders — the first two quarters of production telemetry replace them, and the NFR register versions that change.
8 · Backup / DR operations & drill calendar
Operations follow RB-17: policy-driven backups with immutability on T1 sets, isolated cyber vault with a separate credential realm, restore tests that measure timings rather than assert success, tier-ordered failover rehearsals using the recovery path lens, and failback with delta reconciliation. The standing calendar (cadences placeholder until BIA):
| Drill | Scope | Cadence (placeholder) | Evidence |
|---|---|---|---|
| Restore test | Each T1/T2 store, rotating restore points | Per tier cadence | Timed report + integrity checks |
| DR failover rehearsal | Tier-1 service chain end-to-end | Semi-annual placeholder | Observed RTO vs band; gap actions |
| Cyber-vault recovery | T1 recovery set from vault, hostile-primary assumption | Annual+ placeholder | Timed report; isolation verification |
| Paging/escalation drill | On-call chain incl. executive bridge | Quarterly placeholder | Time-to-engage record |
| Degradation-matrix exercise | Vendor-outage stances per RB-21 | Annual placeholder | Matrix accuracy review |
Regional failover is deliberately absent from this calendar until OD-01 decides the posture — rehearsing a strategy that hasn't been chosen would manufacture false confidence. The interim truth (backups + rebuild for regional loss) is stated in the resilience view.
9 · AIOps stance
AIOps (ops-aiops) is optional and hygiene-gated: correlation and anomaly detection are adopted only after alert ownership, SLO discipline and telemetry quality are proven — machine-correlating noisy, unowned alerts automates confusion. When adopted, its recommendations attach to incidents as suggestions with confidence context; humans approve any action; every suggestion and outcome is audited so its precision is measurable. Rollback of an AIOps-suggested change follows the same GitOps revert as any change. Production-affecting autonomy for AIOps would require the same graduated-autonomy treatment as business agents (ADR-14 pattern) — none is granted in this architecture.
10 · Telemetry retention, privacy & cost
Telemetry is data too: collectors redact secrets and personal data before storage (privacy overlay applies to logs — C-PRV-01 posture); retention tiers by signal class (traces shortest, metrics longest, security logs per C-GOV-03) with archival for the few classes that need it; sampling strategies (head + tail for traces) keep fidelity where incidents live. Ingest cost is a monthly FinOps line with a per-service unit view — observability spend growing faster than traffic is a design smell flagged at the reliability review. The cost lever is sampling and retention policy, never "stop instrumenting".