SLO mechanics, observability standards, incident and problem management, capacity, backup/recovery operations, AIOps stance, and telemetry cost/privacy controls. Companion views: Observability & SRE and Resilience & DR; procedures live in the runbook library.
Honesty header: every numeric target below is a placeholder band until the BIA (P1) and volume data (OQ-04) exist — the framework is real, the numbers are not yet (A-05, A-10).
Every service in the catalog declares its SLIs from a standard menu — availability (good requests / total), latency (p95/p99 against its NFR band), quality (error class rates), freshness (data products), lag (event consumers) — and sets SLOs per its service tier at onboarding (RB-05). The error budget is the operational contract: while budget remains, teams ship; when a budget burns down within its window, the published error-budget policy activates — feature work yields capacity to reliability work, risky deploys pause, and the reliability review (gov-reliability) tracks the recovery. Budgets make the reliability/velocity trade explicit instead of political (S37 formalized; thresholds placeholder until tiers are BIA-confirmed).
| Tier | SLO stance (placeholder bands — BIA binds numbers) | Review cadence |
|---|---|---|
| T1 mission-critical | Tightest availability/latency SLOs; multi-AZ mandatory; synthetic coverage mandatory | Monthly reliability review + per-incident |
| T2 business-critical | Strong SLOs; multi-AZ mandatory; synthetics on key journeys | Monthly |
| T3 important | Standard SLOs; single-region multi-AZ preferred | Quarterly |
| T4 supporting | Best-effort with defined support hours; vendor SLAs where SaaS | Quarterly |
OpenTelemetry is the instrumentation standard (ADR-11): SDKs with enterprise semantic conventions (service.name from the catalog, environment, tier, owner labels), collectors per cluster/zone performing sampling and redaction, one consolidated backend for metrics, logs and traces. Correlation ids propagate end-to-end — channel → gateway → BFF → services → events (trace context in headers and event metadata) → data pipelines — so a customer complaint can be walked to the exact failing hop. Service maps derive from real trace topology, not diagrams; the canonical model and the observed map are reconciled quarterly (drift between them is a finding). Golden-path templates ship instrumentation pre-wired, which is why coverage is an adoption metric rather than a campaign.
Three rules: every alert has a named owning team (orphan alerts are deleted or adopted at the monthly audit); alerts page on symptoms and SLO burn, not raw causes (cause-level signals become dashboard context); and every page links a runbook or it doesn't page. Routing follows catalog ownership through the paging service (ops-incident) with tiered urgency: page (now), ticket (business hours), FYI (digest). Noise is managed as debt — the alert-quality review tracks pages-per-shift and actionability rate, and noisy rules lose their paging rights until fixed.
Incidents run per RB-23: severity matrix, incident commander at Sev-2+ (command separate from keyboard), stabilize-first doctrine (rollback / failover / degrade / scale before root-causing), honest comms on cadence, and verification via synthetics before closure. Security suspicion pulls in RB-16 immediately — availability restoration must not destroy evidence. Problem management is the follow-through: blameless post-incident reviews within days placeholder produce actions with owners and dates; repeat incidents from unactioned PIR items are a reliability-review escalation, because a lesson not implemented is a lesson not learned. Trends (by service, by cause class) feed the risk register and the roadmap's optimization backlog.
Teams own the pager for what they build (domain services, data products, platform components alike). Rotas are staffed to a humane density placeholder; escalation ladders are short and named: primary → secondary → team lead → duty manager → duty executive (crisis). Platform and security maintain their own 24×7-capable rotas placeholder pending org sizing. Every on-call joiner completes a readiness checklist (access verified via PAM, runbooks read, shadow shift done). Pager health is reviewed monthly: page volume, sleep-hours pages, handoff quality — sustained overload is treated as a capacity or architecture defect, not a personal one.
Standard changes flow through golden-path pipelines with GitOps delivery (RB-13) — the pipeline evidence is the change record for low-risk classes, keeping the ITSM overhead proportional to risk. Higher-risk changes add explicit approval and verification plans; freeze windows protect cutovers and peak periods with RB-14 as the only exception path. Configuration truth lives in Git (IaC + manifests) reconciled by controllers; the CMDB holds service-level records (ownership, tier, dependencies) synchronized from the catalog and this package's model — drift between deployed reality, Git and the model is detected (C-CLD-02) and treated as an incident, which is what keeps the architecture repository honest over time.
Capacity runs on telemetry, not folklore (RB-18): each component's capacity record (in the model) names its scaling triggers, owner and test method; dashboards project utilization against growth assumptions; reviews happen on cadence and before known load events, with pre-scale/post-scale-down discipline enforced by FinOps checks. Autoscaling is bounded by quotas so a bug cannot buy infinite compute. Until OQ-04 volumes exist, all thresholds are placeholders — the first two quarters of production telemetry replace them, and the NFR register versions that change.
Operations follow RB-17: policy-driven backups with immutability on T1 sets, isolated cyber vault with a separate credential realm, restore tests that measure timings rather than assert success, tier-ordered failover rehearsals using the recovery path lens, and failback with delta reconciliation. The standing calendar (cadences placeholder until BIA):
| Drill | Scope | Cadence (placeholder) | Evidence |
|---|---|---|---|
| Restore test | Each T1/T2 store, rotating restore points | Per tier cadence | Timed report + integrity checks |
| DR failover rehearsal | Tier-1 service chain end-to-end | Semi-annual placeholder | Observed RTO vs band; gap actions |
| Cyber-vault recovery | T1 recovery set from vault, hostile-primary assumption | Annual+ placeholder | Timed report; isolation verification |
| Paging/escalation drill | On-call chain incl. executive bridge | Quarterly placeholder | Time-to-engage record |
| Degradation-matrix exercise | Vendor-outage stances per RB-21 | Annual placeholder | Matrix accuracy review |
Regional failover is deliberately absent from this calendar until OD-01 decides the posture — rehearsing a strategy that hasn't been chosen would manufacture false confidence. The interim truth (backups + rebuild for regional loss) is stated in the resilience view.
AIOps (ops-aiops) is optional and hygiene-gated: correlation and anomaly detection are adopted only after alert ownership, SLO discipline and telemetry quality are proven — machine-correlating noisy, unowned alerts automates confusion. When adopted, its recommendations attach to incidents as suggestions with confidence context; humans approve any action; every suggestion and outcome is audited so its precision is measurable. Rollback of an AIOps-suggested change follows the same GitOps revert as any change. Production-affecting autonomy for AIOps would require the same graduated-autonomy treatment as business agents (ADR-14 pattern) — none is granted in this architecture.
Telemetry is data too: collectors redact secrets and personal data before storage (privacy overlay applies to logs — C-PRV-01 posture); retention tiers by signal class (traces shortest, metrics longest, security logs per C-GOV-03) with archival for the few classes that need it; sampling strategies (head + tail for traces) keep fidelity where incidents live. Ingest cost is a monthly FinOps line with a per-service unit view — observability spend growing faster than traffic is a design smell flagged at the reliability review. The cost lever is sampling and retention policy, never "stop instrumenting".