Runtime feature flags staging rehearsal plan and production prerequisites¶
This page holds two written plans from the remaining delivery plan:
- the staging rehearsal plan (6C.3), with entry and exit criteria for Chris's later gate;
- the production prerequisites list (6C.4), with an approval record template.
Every step points to a runbook step in Operate runtime feature flags
(R<runbook>.<step>). The local rehearsal that comes first is in the
isolated rehearsal checklist.
Roll-forward means the next reviewed forward change if a step fails or must be reversed. Deployments and data only move forward (ADR-023); a rollback is a planned replacement release.
Staging rehearsal plan (6C.3)¶
NOT EXECUTED. This is a plan. No step below has been run in staging, in a PR preview or in production. Staging execution needs Chris's explicit decision (owner gate, delivery plan §8).
Scope¶
- One control:
disableMembership(D2). Statistics keys are not enrolled as runtime decisions. - API and PM are promoted together (D4).
- The rehearsal proves the mechanism: bind, import, enrol, collect, change, stop, restore and the Legacy-return checks. It is evidence about the mechanism, never authorisation for production.
- The RT-69 part of OD-26 (guide page and What's New entry appearing at the moment of the switch) runs as the
RT-69 rows below, on a release of the rehearsal control. It needs
RuntimeFeatureReleases:EnabledandRuntimeFeatureReleases:OutboxDispatch:Enabledon in staging, which is a separate owner decision (delivery plan §8); until then those rows are recorded as outstanding.
Stage 0: throwaway PR preview (D1)¶
Before staging, the same sequence may run in a throwaway PR preview with its own database and no real users, then be torn down. Gaps that block it today:
- previews do not run
bindorup, so migrations0009–0021are never applied there (runbook 1 preconditions); - the lineage command has no in-cluster delivery mechanism (runbook 2);
- the machine-client secrets have no chart wiring on Identity (runbook 4).
Until those are resolved, Stage 0 is limited to what the isolated rehearsal covers.
Entry criteria for Chris's gate¶
Each criterion names how it is verified.
- 6A and 6B are merged (#4407, server-time retirement, merged as
9174f9e9). Verify: PR states. - The isolated rehearsal (6C.2) transcript shows every row of the checklist as expected, or as an accepted gap. Verify: the stored evidence directory.
- Staging's API and PM run the same
mainbuild, which contains migrations0009–0021. Verify:/health/liveversions andmigrations.lock.jsonat that commit. - The pre-deploy migration Job can be enabled in staging (pre-deploy plan R1). Verify: the reviewed cluster-gitops change.
- The known gaps that block a step are resolved or explicitly accepted by Chris:
status/verifydelivery, lineage delivery, Identity secret wiring, collector scheduler, writer freeze mechanism. All 16 items are tracked in #4403; items 14 and 15 wait on owner decisions D7 and D8 on the programme progress page. Verify: each item has a linked PR or a recorded decision. - A staging read-only Kubernetes identity for the collector exists (precedent
syrf-stats-preflight). Verify: its role definition reference. - No active deployment constraint and no other runtime-policy change is planned during the rehearsal window.
Verify: report
deploymentConstraintprojection isnull. - A rehearsal window and a named operator are recorded in the approval record. Verify: the record.
Steps¶
| Step | Runbook ref | Owner | Approver | Evidence | Roll-forward |
|---|---|---|---|---|---|
| S1 Read the live fingerprint | R1.1 | Operator | Chris | status output |
Read-only; none needed |
| S2 Enable the migration Job and bind | R1.2–R1.4 | Operator | Chris (cluster-gitops PR) | PR URL; bind and up logs with epoch |
A failed PreSync keeps the previous release serving; fix forward with a new commit. A wrong binding cannot be replaced (rebind not built): stop and escalate |
| S3 Verify migrations | R1.5 | Operator | n/a | verify output or report migrations: Ready |
Forward migration fix in a new release |
| S4 Provision the report and collector clients | R4.1–R4.2, R7.3 | Operator | Chris (secrets owner) | Client registration change (no secrets) | Revoke by removing the secret (Identity revokes the client) |
| S5 Run the collector for API and PM | R4.3–R4.6 | Operator | n/a | Capture files and receipts | Replay the exact capture on exit 3; new capture otherwise |
| S6 Review the baseline JSON | R2.1 | Operator | Chris | Baseline file, approval, provenance | Edit and re-review before S7 |
| S7 Lineage preview | R2.2–R2.3 | Operator | Chris (reviews the file) | Preview file and digest | Discard; make a new preview |
| S8 Freeze and scale the API to zero | R2.4 | Operator | Chris | Freeze reference, zero-pod proof | Restore replicas with a forward change if S9 is abandoned before apply |
| S9 Lineage apply | R2.5–R2.6 | Operator | Chris | Receipt file, logs | No undo. Correct values with a reviewed policy revision (S13). A wrong lineage needs rebind (not built) |
| S10 Enrol API and PM with inventory and Collector mode | R3.1–R3.3, R6.1–R6.3 | Operator | Chris (cluster-gitops PR) | PR URL, rendered env diff | Forward values fix. Never return the API to Legacy (it refuses the imported authority) |
| S11 Resume the manual capture-and-publish loop (cycle under 30 s), post-enrolment checks; end the freeze | R3.4 | Operator | n/a | Loop receipts; authority-status, authority-evaluation, fleet responses (Unknown with a Collector… reason is expected until fresh observations land) |
Runbook 9 cases |
| S12 Policy-change report and checker | R7.3 | Operator | n/a | Report file, checker output | Fresh report |
S13 Flip disableMembership on |
R7.1–R7.6 | Operator | Chris | Preview, receipt, fleet Active timing | Reviewed forward revision (off) or stop (S15) |
| S14 Flip it off | R7.1–R7.6 | Operator | Chris | As S13 | Reviewed forward revision |
| S15 Emergency stop | R8.1–R8.3 | Operator | Chris | Stop receipt, per-host adoption | Restore (S16) |
| S16 Restore | R8.4–R8.5 | Operator | Chris | Preview, receipt, fleet Active | Another stop |
| S17 Lost-response drill | R7.7, R9.1 | Operator | n/a | Lookup result; exact replay returns the original receipt | None needed |
| S18 Rollout drill: one ordinary API and PM rollout | R6 notes | Operator | Chris | Fleet states during rollout; old rows Retired; Active again |
Forward change |
| S19 Legacy-return checks only (no mode switch) | R10.1–R10.3 | Operator | n/a | Report legacyParity, checker output |
Read-only. R10.4 is executable since #4414 but is not part of this rehearsal |
| S20 Optional: coordinator record, extend, release | R5.1–R5.4 | Operator | Chris | Receipts, report before and after | Release, or break-glass (S21) |
| S21 Optional: break-glass drill | R9.6 | Operator (human administrator) | Chris | Reason, receipt, report | None needed |
| S22 Store the evidence and approval record | All | Operator | Chris | Run record | n/a |
RT-69 release guide and What's New check (6G)¶
Accepted plan RT-68 and RT-69 (OD-16, OD-26): switching a release on makes its guide page and What's New entry available to its audience at the same moment; a stop hides the entry (the guide link stays, the in-app chip shows "Currently unavailable" once 6G.5 E2 ships); a restore brings it back with its original date; reaching general release publishes it on the public What's New dated that day. Each row stores captured responses and screenshots, before and after the step, with the release record.
Captured responses, every row:
GET /api/projects/{projectId}/features/availabilityas a project administrator, andGET /api/features/availability(no project) as the same person and as an ordinary member: the feature'savailability,release,announcementandwhatsNew.GET /api/whats-new/publicanonymously (no cookie, no token;curl -sS -D -), keeping the headers:Cache-Control: public, max-age=60and the body. It is the same for every caller and carries published content only (D26 A).
Screenshots, every row: the home page signed out, the home page and the project overview ("New in this project")
signed in as the administrator, light and dark. Staging delivers announcements exactly as production does (D60 A,
26 Oct 2026), so a delivered entry shows "New to you" and nothing is labelled "Rehearsal"; the staging binding keeps
its stored testSink value, which is no longer read, so no re-bind is needed.
| Step | Runbook ref | Owner | Approver | Evidence and expected result | Roll-forward |
|---|---|---|---|---|---|
| RT69-1 Before: release prepared, not applied | 6D prepare | Operator | Chris | Availability: no release, whatsNew: null. Public: entries: []. Home page signed out identical to before 6G |
None needed |
RT69-2 Apply as LimitedRelease and wait for confirmation and the MaterialiseWhatsNew row Done |
6D apply, R7 | Operator | Chris | Administrator in the project: whatsNew present, dated by the LimitedRelease milestone; ordinary member and no-project answers: no entry. Public: still entries: [] (a limited release is never public). Guide page reachable from "Read the guide" |
Emergency stop (S15) |
RT69-3 Advance to GeneralRelease and wait for confirmation and its row Done |
6D apply, R7 | Operator | Chris | Public: one entry, releasedAtUtc equal to the GeneralRelease milestone, announcementId {releaseId}:GeneralRelease equal to the signed-in answer's announcement.id. Signed out the home page shows it; signed in it shows once |
Emergency stop (S15) |
| RT69-4 Emergency stop | R8.1–R8.3 | Operator | Chris | Within 60 s plus the 10 s release read: the administrator's answer is unavailable/stopped with whatsNew: null, and a fresh public request (no cache) has no entry. A shared cache may serve the old public body for up to 60 s; record the time it cleared |
Restore (RT69-5) |
| RT69-5 Restore | R8.4–R8.5 | Operator | Chris | The public body is byte-identical to RT69-3's (same date); the signed-in entry returns with the same releasedAtUtc; no "New to you" |
Another stop |
RT69-6 Switch-off check: RuntimeFeatureReleases:Enabled=false |
Values change | Operator | Chris (cluster-gitops PR) | Both routes answer 404 {"code":"feature_availability_disabled"}; the home page signed out and signed in is the pre-6G page |
Switch back on (values change) |
Exit criteria¶
- Every executed step has its evidence stored with the run record.
- Each flip (S13, S14) reaches fleet
Activeon its committed identity within the 60-second confirmation window, with per-host adoption times recorded. - The stop (S15) reaches every API decision and every host acknowledgement; restore (S16) brings back the saved override.
- A policy-change report is
Completein Collector mode, and the checker passes it (S12). - The rollout drill (S18) returns to
Activewithout operator action. - The Legacy-return checks (S19) pass.
- No step needed a production credential.
- RT-69 (OD-26): every RT-69 row has its captured responses and screenshots and shows its expected result, or is recorded as outstanding while the release switches stay off in staging.
- Chris reviews the record and decides on production separately.
Production prerequisites (6C.4)¶
NOT EXECUTED. This is a list of prerequisites, not a record of anything done. Every production step needs Chris's approval at the time; merging a production cluster-gitops PR deploys it.
| Step | Runbook ref | Owner | Approver | Evidence | Roll-forward |
|---|---|---|---|---|---|
P1 Catch-up promotion: production on main's runtime-flag code in Legacy mode, API and PM at the same build (D4) |
CD plan, D4 | Release operator | Chris (production PR) | Promotion PR, ArgoCD sync, /health/live versions for both services |
A planned replacement release. The planned rollback to 104eca9 with migration 4's Down is the one built exception (ADR-023) |
| P2 ADR-023 Job enabled in production, with the binding in the same change | R1.1–R1.5 | Release operator | Chris (production PR) | status output, bind and up logs, verify output |
Failed PreSync keeps the previous release; fix forward. A wrong binding needs rebind (not built) |
P3 OD-21 automation identity: Auth0 machine application, scope environment:report:read, credential in a main-only GitHub environment |
R7.3 | Release operator | Chris (production-shared Auth0 tenant change) | Tenant change record (no secrets), API CompatibilityReport settings PR |
Remove the grant or the application |
P4 OD-23 collector: read-only Kubernetes identity and scheduler in camarades-infrastructure, plus an Auth0 machine application for environment:observations:write |
R4.1–R4.6 | Infrastructure operator | Chris (camarades-infrastructure PR, Auth0 change) | Infrastructure PR, first capture and receipt per service | Remove the identity or disable WorkloadCollector |
P5 Coordinator identity and workflow, scope environment:deployments:coordinate |
R5.1–R5.4 | Release operator | Chris | Workflow PR, Auth0 change, first receipts | Disable DeploymentCoordinator; break-glass for a stranded constraint |
P6 Chart and configuration values for API and PM: Mode=Shared, ProductionPolicyMode=Runtime, SharedReaderMode=Disabled, binding, runtimePolicyHostMetadata.enabled, hpa.enabled: false, fixed replicaCount; API also FleetInventoryMode=Collector and both FleetExpectedReplicas |
R3.1, R6.1–R6.4 | Release operator | Chris (production PR) | Rendered env diff, post-rollout checks | Forward values fix, or ProductionPolicyMode=Legacy (runbook 10, once P11 is in the production build) |
| P7 D4 together-deploy and the #4354 mixed-version window | D4 | Release operator | Chris | Both services on one build before any admin step | Finish the rollout; retry after Retry-After |
P8 Collector-mode requirement: production admission follows FleetInventoryMode (since #4407), so set Collector and keep the collector publishing (P4) |
R6.3, R3.4 | Release operator | Chris | A production build containing #4407; a rollout drill showing old rows Retired after the 35-second quiet period (up to 15 minutes for rows from pre-#4407 images) and the fleet Active again |
In Static mode (the default) enables and restores are refused after any rollout; stops still work |
| P9 Initialise the production authority from the deployed values, with the writer freeze (API outage). Production has no legacy policy document (legacy mutation is disabled there), so it uses initialise, not the lineage import (D8=A) | R2.I1–R2.I7 | Release operator | Chris (window, reconciled rendered values, preview) | Input, both previews and the cmp result, reconciled API and PM rendered values, freeze reference, receipt, verify output |
No undo. Correct values with a reviewed forward policy revision; a wrong lineage needs rebind (not built). Needs an in-cluster delivery mechanism first (gap 3). Rehearsed only on an isolated stack (run-local.sh --runtime-fleet-initialise, checklist) |
| P10 Recovery exercise and freshness trial (increment C) | Runbook 9, 6B evidence | Release operator | Chris | Exercise record; measured adoption times | n/a |
| P11 Legacy-return path in the production build: #4414 (D7=A) promoted to production before any import there | Runbook 10 | Developer | Chris | Merged PR, the runtime-fleet legacy-return scenario evidence, and the production build containing it |
n/a |
| P12 First enable as a release (OD-26): guidance under OD-5 and the RT-69 pre-enable check | R7, RT-69 | Product owner | Chris | Guide page and What's New check before and after: the RT-69 rows with production's captured availability and public (GET /api/whats-new/public) responses and screenshots |
Emergency stop (runbook 8) |
P12a Release guidance bundle in the production API image (6D.5, D-Q6; D38 A, 6G.7): deploy an API image built since 6G PR G (#4513), which carries the server-<sha256>.json compiled from the same commit at /app/release-guidance/; leave RuntimeFeatureReleases:GuidanceBundlePath unset. No Secret and no chart value. Without such an image every guidance-required Apply is refused with GuidanceBundleMissing (emergency stops are unaffected) |
Applying a release | Release operator | Chris (production promotion PR) | The image build's "Check release guidance bundles in the image" notice (bundle name and SHA-256), the promoted API image tag, an Apply preview showing guidanceState: Verified with the expected guidanceBundleDigest |
Promote the previous API image (admission then verifies that image's bundle) |
P13 Production enable of disableMembership |
R7.1–R7.6 | Release operator | Chris | Report, checker, preview, receipt, fleet Active | Emergency stop, then a reviewed forward revision |
| P14 Approval record completed for each of P1–P13 (including P12a) | Template below | Release operator | Chris | Stored record | n/a |
Approval record template¶
Copy this block once per step or per rehearsal run. Use placeholders for anything secret.
Approval record
---------------
Step ID(s): <S… or P…>
Environment: <env> (binding epoch <binding-epoch>)
Runbook steps: <R…>
Operator: <name>
Approver: <name>
Approval reference: <chat message, PR approval or issue comment URL>
Window (UTC): <start> to <end>
Freeze reference: <reference, if runbook 2>
Builds: api <image@sha256>, project-management <image@sha256>
Preconditions checked: <list, with the evidence for each>
Commands or requests run: <list; no tokens, no secrets>
Expected output seen: <yes/no per preflight check, with excerpts>
Refusals and handling: <code, action taken, retry mutation IDs>
Evidence stored at: <path or link>
Roll-forward needed: <none / description and PR>
Outcome: <completed / stopped at step …>
Follow-ups: <issue links>