# Rotate credentials

Replace integration credentials through each product's existing seam, with overlap, cache and retry limits made explicit.

Rotate one product and environment at a time. Keep an inventory of the credential's
purpose, its consumers, its public key id or digest, and who can replace it. Store
recoverable secrets only in your secret store or owner-only files, never in this
inventory, a shell argument, a ticket, a URL or a log.

**Rotation is not the same as emergency revocation.** An overlap deliberately keeps
an old credential usable. Public verification keys can remain cached after removal.
Replacing an API key does not cancel requests that have already authenticated or
revoke Cart, Order or preview tokens.

## What can be replaced today

| Credential                                | Existing seam                                                                        | Cutover and limits                                                                                                                     |
| ----------------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| Catalog Source key                        | Connections → New key; staff client `newSourceKey`                                   | Same Source, new hash. Old key stops at commit; no overlap.                                                                            |
| Catalog change-feed read key              | Staff client `addReader` / `removeReader`                                            | Adding the same reader id replaces its key; a second reader id permits overlap.                                                        |
| Catalog event signing key                 | Staff client `rotateSigningKey`                                                      | New key prepublished for 31 minutes; previous public key retained for six minutes after cutover.                                       |
| Catalog webhook secret                    | Staff client `rollWebhookSecret` / `webhookSecret`                                   | New secret immediately; previous secret signs beside it for 24 hours.                                                                  |
| Page event signing key                    | Deployment configuration below                                                       | Publish next/retiring public keys independently of the active private key.                                                             |
| Catalog/Page operator key                 | Offline `orbcatalog operator-key`, then replace/remove `OPERATOR_KEY_HASH`           | No overlap configuration. Old deployments may still accept their old hash until rollout completes.                                     |
| Cart publishable/secret key               | Account-private `CommerceKeys`; `orbcart keys issue/revoke/status --env test`        | Retry-safe local issuance/replacement and terminal digest revocation, with pending/confirmed receipts. Live management is unsupported. |
| Cart event/quote/contract signing key     | Account-private `CommerceKeys`; `orbcart signing prepare/activate/status --env test` | One pending generation; one-day prepublication after all replicas confirm, then explicit activation. Historical public keys remain.    |
| Staff/delegated access token              | Merchant's staff sign-in                                                             | Short-lived tokens, not integration keys; existing tokens need not stop immediately when sessions end.                                 |
| Buyer identity signing key                | Your identity provider's JWKS or deployed inline public keys                         | Provider-owned lifecycle; inline keys require deployment.                                                                              |
| Provider and outbound integration secrets | Provider's own lifecycle and the product's private configuration                     | Issue with the provider, update every consumer, then revoke there. No family-wide overlap promise.                                     |

Catalog Source and reader lookups require fresh database reads: keep the deployed
Hyperdrive query cache disabled as configured. Do not turn a database cache into an
authentication cache. Use the roles the current product contract requires; an operator
setup credential is not a general staff or owner credential.

## Catalog Source and reader keys

For a Source, pause its sync and reconcile in-flight writes first. Call `newSourceKey`
for the existing Source id with one stable idempotency key, securely retain the returned
secret, update the integrator and resume. Do not create a second Source as a substitute
for replacing its key. Verify a new authenticated request succeeds and a request with
the old key fails. Removing the Source also relinquishes its field ownership, so it
is not the way to rotate its key.

Source issuance/replacement responses expose the key only once. An identical retry
does not recover the secret. If the response was lost, explicitly replace the key
again with a new idempotency key, then install that replacement. Do not record raw
keys in a retry log. See [Catalog Sources](/docs/integrate/catalog-sources).

For feed readers, save the consumer's last processed cursor before changing keys.
Calling `addReader` with its existing id replaces the key without overlap. To overlap,
add a reader with another id, install that key in the same consumer while preserving
its cursor, verify progress, then remove the old reader. Keep processing/deduplication
state independent of the credential. Removing the old reader disables its key; it
does not undo events the consumer has processed.

## Catalog signing keys and webhook secrets

Use `rotateSigningKey` with a stable idempotency key. Record the returned key ids and
`signsFrom` timestamp, not secret material. A pending generation answers `key_waiting`
instead of adding another waiting key. Wait until `signsFrom`, verify a delivery under
the new key, and allow the previous key's six-minute verification window to finish.
These are the Catalog's cache-aware timings, not timings to reuse for Cart quotes.

For a webhook secret, use `rollWebhookSecret` once and install the returned current
secret at the receiver. Keep verifying the previous secret during its returned
`previousUntil` window. Test the receiver before retiring its old configuration.
If the answer was lost, retry with the same idempotency key and read the current secret
through `webhookSecret` if needed. Do not issue another independent roll as a retry:
a second roll drops the older previous generation, shortening its overlap.

The 24-hour window is an availability feature, not immediate revocation of a leaked
secret. Receiver configuration and third-party caches remain your responsibility.

## Page event signing keys

Page has no public integration API-key type. Staff tokens authorize its management
API; an operator credential authorizes only its allowed setup operations. The event
key signs outgoing form notifications and `content.published` events, not API calls
from an integrator to Page.

`EVENT_SIGNING_KEY` contains the active ES256 private key in PKCS #8, base64. Generate
its replacement in your secret store and retain it there. `EVENT_VERIFICATION_KEYS`
contains a JSON JWKS of public ES256 keys only, with these required fields per key:
`kty`, `crv`, `x`, `y`, `kid`, `alg: "ES256"`, `use: "sig"`. Use the JWK thumbprint
as `kid`, matching the signing key's published id. Never paste the private JWK's `d`
or a sealed/private key into this setting.

Page publishes the active public key plus the configured verification keys at
`/.well-known/jwks.json`, with `Cache-Control: no-store`. Exact duplicate keys are
deduplicated by `kid`. Conflicting key material for one id, private members, malformed
JSON and invalid curve points refuse publication without exposing the configuration.
Verification keys can be published even when no active private signer is configured.

For ordinary replacement:

1. Deploy `EVENT_VERIFICATION_KEYS` containing both the old and next public keys,
   leaving `EVENT_SIGNING_KEY` unchanged. Finish the entire rollout and confirm the
   public JWKS contains both ids.
2. Wait at least five minutes after that rollout, so the family's HTTP-signature
   verifiers can refresh. If a receiver uses a longer cache, wait for that too.
3. Switch `EVENT_SIGNING_KEY` to the prepared replacement, keeping both public keys
   published. Complete the rollout; verify a fresh delivery carries the new id.
4. Keep the old public key for at least six minutes after the last old deployment
   stops signing, longer if your supported receiver's signature lifetime requires it.
   Then remove it from `EVENT_VERIFICATION_KEYS` and complete that rollout.
5. Verify new deliveries still succeed. After verifier caches refresh, a freshly
   signed synthetic request under the retired key must fail.

On retry, both Page event paths keep the event id and JSON body but read the current
private signer and create a new signature/timestamp. Deduplicate by source and event
id, not signature or signing-key id. Do not replay a stored HTTP signature after its
five-minute validity window. A rollback to a deployment containing the old private key
can resume old signing: keep the overlap until rollback is no longer possible, or
roll forward with the replacement key instead.

## Cache freshness and retry behavior

The family's HTTP-signature verifier trusts a successfully fetched JWKS for five
minutes. An unknown key may refresh it, at most once a minute. Concurrent fetches are
coalesced. A failed fetch does **not** renew those five minutes: an expired remembered
key answers `keys_unavailable` during an outage, with another attempt allowed after
one minute. A successful refresh removes keys absent from the new set.

Return a retryable failure, such as HTTP `503`, for `keys_unavailable`; do not process
or acknowledge the event. Invalid signatures are refused. Retry the unchanged event
later and deduplicate it normally. A long JWKS outage therefore interrupts delivery
rather than extending trust in removed keys indefinitely. Endpoint/proxy caches add
their own lifetime; the verifier cannot purge them or other implementations' caches.

JWT verification is separate. The family uses JOSE's ten-minute remote-key cache,
30-second unknown-key cooldown and five-second fetch timeout. Staff tokens normally
last 15 minutes with a 60-second clock tolerance. Delegated tokens last five minutes
with no expiry tolerance; refresh rechecks membership. Ending sessions or removing
membership prevents further issuance, not immediate rejection of already issued
offline-verifiable tokens. Buyer token expiry and key publication are issuer-owned.

## Cart API-key replacement and revocation

The commands below and the `CommerceKeys` entrypoint deliberately refuse environments
other than `test`. Run from the merchant's Worker directory, with account access to
its private entrypoint. `--config` and `--worker` select a nondefault configuration or
Worker. There is no public HTTP management endpoint: a stolen secret API key cannot
mint, revoke or rotate keys.
Complete the runtime rollout before using these operations: an older deployment
that still knows only issuance cannot enforce the new revocation state.

Issue replacements into a **new file**, retaining the old file until cutover:

```sh
orbcart keys issue --env test --channel de --keys .orbcart/keys/test-de-next.env
```

This saves both random replacement keys in the owner-only file, synchronizes the
file and its directory, and only then registers their SHA-256 digests. The server
keeps no recoverable API-key copies. A retry with the **same path and channel** reuses
the saved secrets and digests, including after an unknown/lost response; it does not
issue another generation. Protect that file as a secret, and do not overwrite or
delete it to retry. An existing file with terminally revoked keys is not silently
replaced: choose a new replacement file instead.

The command prints secret-free receipts with `digest`, `state`, `version` and
`status`. `state: registered` or `state: revoked` is the writer's durable state;
`status: pending` means the change is committed but not confirmed by every replica.
`status: confirmed` acknowledges the returned version on every replica. The issuance
command exits nonzero until both registrations are confirmed, or if either digest
has been revoked. Do not install a merely saved/pending key as a completed cutover.
A digest already bound to different permissions answers `status: conflict`, without
changing those permissions; the command exits nonzero. Use the correct original
channel, or a new replacement file, rather than trying to repurpose that digest.

After confirmed registration, install the replacements in their consumers and prove
fresh requests succeed. Issuing replacements does not revoke the originals. Retain
the old keys' digests from their receipts, then revoke each original digest:

```sh
orbcart keys revoke --env test --digest <old-digest>
orbcart keys status --env test --digest <old-digest>
```

Retry the same digest until the receipt says `state: revoked, status: confirmed`,
then prove an old-key request fails and the replacement still works. Registration
and revocation retries do not add another version. Revocation is terminal, even if
it precedes a delayed registration, and tombstones survive the retained history's
1,000-change floor. Unknown status returns `not_found`; it is not proof of revocation.

Replicas catch up once on every cold start before authentication or signing,
including a restored **nonzero** snapshot; failure to reach the writer prevents
that cold instance from serving. Warm replicas use asynchronous pushes rather than
a writer read on every shopper request. A **pending** operation has no fixed deadline
during a partition: an already warm, isolated replica may still recognize an old
key. Confirmation does not cancel requests that already authenticated.

Only one CLI process may write a key file at a time; a sibling `.lock` file refuses
concurrent commands. If a process crashes and leaves the lock, first ensure no
command is still using that path, remove only the stale lock, and retry with the
unchanged key file. Never remove the secret file as lock recovery.

Before a Cart API-key cutover, drain and reconcile ambiguous writes. Cart's
idempotency entries are scoped to the credential digest: repeating the same
`Idempotency-Key` under a different API key is not guaranteed to replay the earlier
result. Cart/Order bearer tokens and Page's 15-minute preview tokens have separate
owners and lifetimes; replacing an integration key does not revoke them.

## Cart signing-key rotation

```sh
orbcart signing prepare --env test
orbcart signing status --env test --generation <returned-generation>
orbcart signing activate --env test --generation <returned-generation>
```

`prepare` keeps the active signer unchanged and publishes one next generation in
JWKS. Retrying preparation while that generation is pending returns the same id,
including while its activation is still propagating. Record the public `generation`
and receipt, never the sealed private key or `ROOT_SECRET`.

- `publishing`: publication is committed, but a replica has not confirmed it;
  `publishedAt` and `activatesFrom` remain null. Waiting a day from preparation
  does not bypass this barrier.
- `waiting`: all replicas confirmed publication at `publishedAt` (Unix milliseconds).
  `activatesFrom` is exactly one day later. This full day starts at confirmation,
  not at preparation or the first replica's publication.
- `ready`: that day has elapsed, but the old key **still signs** until an explicit
  `activate` for this exact generation. Early activation returns the waiting receipt
  without changing the signer.
- `activating`: activation is committed and some replicas may already sign with
  the new key; it is not yet acknowledged everywhere. Both public keys were already
  published for the full day. Retry the same generation or inspect status.
- `active`: the activation version has been confirmed everywhere. Later retries
  report completion of that generation's activation; they do not reactivate an old
  generation or replace a newer signer.

Preparation permits only one unfinished generation. Activation never chooses an
implicit "latest" generation and never creates one. Private keys remain sealed by
the unchanged root. Old public keys remain in JWKS for historical quote verification;
there is no ordinary-rotation operation to remove them. This is availability-safe
replacement, **not emergency revocation of a compromised signing key**. Local tests
cover quote verification, asynchronous publication/activation, retries and restart;
installed/deployed and receiver-cache acceptance still require authorization below.

## Never rotate a sealing root as a routine key change

Catalog's `SIGNING_KEY` is both its first signing key and the source of the seal for
later keys and other sealed credentials. Cart's `ROOT_SECRET` derives signing-key
encryption, buyer-index names, gift-card functions and handoff signatures. Blindly
replacing either can make stored state unreadable or unreachable. A root compromise
requires a separately designed and authorized recovery/migration, not this runbook's
ordinary replacement procedure. Retaining old public keys for valid quotes is also
not emergency revocation of a compromised signing key.

## Verify an installation

Local synthetic tests are not deployed acceptance. With explicit authorization for
the intended installation, record product/environment, public ids or digests,
rollout completion and overlap times, old rejection/new acceptance, warm-cache
verification, receiver outage/retry, and restart/recovery results. Keep raw keys,
provider responses and personal data out of the evidence. Installed acceptance stays
open until those authorized checks run; a configured address or a green local test
does not establish it.