Key Rotation Runbook

This runbook covers scheduled rotation of cryptographic keys and certificates in a Mantis deployment. Regular rotation limits the exposure window if a key is compromised.

Rotation Schedule

Key Material	Recommended Interval	Trigger
Encryption master key	Annually	Schedule or compromise
TLS certificates	Annually (or before expiry)	Schedule or expiry
CA certificate	Every 3-5 years (deliberately earlier than the `mantisctl cert generate-ca` default of 3650 days / 10 years — issue with `--days 1825` if you want the cert to expire on this cadence)	Schedule
JWT signing key (RS256 PEM pair, or HS256 secret)	Annually	Schedule or compromise
API keys	Per organizational policy	User-initiated

Pre-Rotation Checklist

Before performing any rotation:

Verify current backups are recent and valid
Schedule a maintenance window (rotation may cause brief service interruptions)
Notify affected teams
Have rollback plan ready (old keys accessible)
Test the procedure in a non-production environment first

Encryption Master Key Rotation

The master key is used directly as the AES-256-GCM key for credentials, secrets and sensitive variables — there is no key derivation (no HKDF/PBKDF/Argon). Separation between records comes from a random 96-bit nonce plus record-scoped additional authenticated data.

Procedure

# 1. Generate new master key
NEW_KEY=$(openssl rand -base64 32)
echo "New key generated (store securely before proceeding)"

# 2. Back up the current key
echo "$MANTIS_ENCRYPTION_KEY" > /secure/old-master-key.bak
chmod 600 /secure/old-master-key.bak

# 3. Run key rotation. This connects to PostgreSQL directly and requires
#    DATABASE_URL -- there is no --database-url flag.
export DATABASE_URL="postgres://mantis:...@db-host:5432/mantis"
mantisctl key rotate \
  --old-key "$MANTIS_ENCRYPTION_KEY" \
  --new-key "$NEW_KEY"

# 5. Update secrets manager with new key
vault kv put secret/mantis/encryption key="$NEW_KEY"
# or
aws secretsmanager update-secret --secret-id mantis-encryption-key \
  --secret-string "$NEW_KEY"

# 6. Update environment and restart services
# systemd reads /etc/mantis/{mandible,thorax}.env, not the shell environment
for svc in mandible thorax; do
  sed -i "/^MANTIS_ENCRYPTION_KEY=/d" /etc/mantis/${svc}.env
  echo "MANTIS_ENCRYPTION_KEY=$NEW_KEY" >> /etc/mantis/${svc}.env
done
systemctl daemon-reload
systemctl restart mandible thorax

# 7. Verify services are healthy
curl -s https://localhost:3000/api/v1/health | jq

# 8. After confirming everything works, securely delete old key backup
shred -u /secure/old-master-key.bak

Rollback

If rotation fails midway:

The rotation processes records in batches. Each batch commits independently, so records rotated before an interruption remain on the new key. If interrupted mid-run, a checkpoint is saved and the process can be resumed with mantisctl key rotate --resume. A record that fails to rotate aborts the rest of that table for the run (the failure is logged and the table stops); resume with --resume after fixing the cause.
If the service fails to start with the new key, revert to the old key
Restart services with the old key and investigate the failure

TLS Certificate Rotation

Component Certificates (Mandible, Thorax, Tarsus)

Use mantisctl cert generate-server / generate-client, which set the SANs and EKUs the platform requires. A hand-rolled openssl x509 -req drops both, and rustls/webpki matches on SAN only (no CN fallback) and enforces the EKU, so the certificate is rejected with NotValidForName / UnsupportedCertificate.

# Server identity (Thorax listener + Mandible internal gRPC share it): needs SANs
mantisctl cert generate-server \
  --name server \
  --sans thorax --sans mandible --sans <thorax-host> --sans <mandible-host> \
  --output-dir /etc/mantis/certs

# The dispatch identity MUST carry CN "mandible" -- it is the system principal
# that tenant-scoped dispatch trusts; a CN of "dispatch" is not trusted.
mantisctl cert generate-client \
  --name dispatch --cn mandible \
  --output-dir /etc/mantis/certs

# 3. Back up old certificates
cp /etc/mantis/certs/mandible-cert.pem /etc/mantis/certs/mandible-cert.pem.bak
cp /etc/mantis/certs/mandible-key.pem /etc/mantis/certs/mandible-key.pem.bak

# 4. Replace certificates
mv /etc/mantis/certs/mandible-new-cert.pem /etc/mantis/certs/mandible-cert.pem
mv /etc/mantis/certs/mandible-new-key.pem /etc/mantis/certs/mandible-key.pem
chmod 600 /etc/mantis/certs/mandible-key.pem

# 5. Restart the component
systemctl restart mandible

# 6. Verify connectivity
curl -s https://localhost:3000/api/v1/health | jq
# /health/grpc discloses the overall status (healthy/degraded/unhealthy) to
# anyone, but the breaker internals (circuit_state, failure_count,
# success_count) are only returned to authenticated callers -- anonymous
# requests get an empty circuit_state and zeroed counts. Pass a token to see
# the breaker detail:
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:3000/api/v1/health/grpc | jq

Rotate All Components

When rotating all certificates (e.g., annual rotation), rotate in this order to minimize disruption:

Tarsus agents — They connect to both Mandible (register/heartbeat) and their assigned Thorax instance (command polling and result submission in poll mode; Thorax connects back to Tarsus in listen mode). Rotate Tarsus certificates before Thorax so the polling connection is re-established against a known-good Thorax.
Thorax — Restart after Tarsus to maintain execution capability
Mandible — Restart last; it coordinates everything

Each component restart briefly interrupts its connections, but the circuit breaker and reconnection logic handle transient failures.

JWT Signing Key Rotation

Mandible signs JWTs with one of two algorithms ([jwt] algorithm):

RS256 (the production default) — an asymmetric RSA key pair on disk (private_key_path / public_key_path). The Ansible deployment generates /etc/mantis/keys/jwt-private.pem and /etc/mantis/keys/jwt-public.pem.
HS256 — a single shared symmetric secret (32+ chars), set via MANDIBLE__JWT__SECRET.

Both support a graceful overlap window via the previous_* settings, so existing tokens stay valid until they expire instead of logging everyone out at once.

RS256 PEM Key-Pair Rotation (production)

# 1. Generate a new RSA private key and derive its public key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
  -out /etc/mantis/keys/jwt-private-new.pem
openssl rsa -in /etc/mantis/keys/jwt-private-new.pem -pubout \
  -out /etc/mantis/keys/jwt-public-new.pem
chmod 600 /etc/mantis/keys/jwt-private-new.pem

# 2. Keep the OLD public key as `previous_public_key_path` so already-issued
#    tokens still validate during the overlap window. In mandible.toml:
#    [jwt]
#    algorithm = "RS256"
#    private_key_path = "/etc/mantis/keys/jwt-private.pem"
#    public_key_path  = "/etc/mantis/keys/jwt-public.pem"
#    previous_public_key_path = "/etc/mantis/keys/jwt-public-old.pem"
#    previous_key_id = "<old-key-id>"

# 3. Promote the new keys (back up the current pair first)
cp /etc/mantis/keys/jwt-public.pem  /etc/mantis/keys/jwt-public-old.pem
cp /etc/mantis/keys/jwt-private.pem /etc/mantis/keys/jwt-private.pem.bak
mv /etc/mantis/keys/jwt-private-new.pem /etc/mantis/keys/jwt-private.pem
mv /etc/mantis/keys/jwt-public-new.pem  /etc/mantis/keys/jwt-public.pem

# 4. Restart Mandible; new tokens are signed with the new key, old tokens still
#    validate against previous_public_key_path until they expire
systemctl restart mandible

# 5. After the longest token lifetime has elapsed, remove the previous_* settings
#    and delete jwt-public-old.pem / jwt-private.pem.bak

HS256 Secret Rotation

# 1. Generate new JWT secret
NEW_JWT_SECRET=$(openssl rand -base64 64)

# 2. Move the current secret to `previous_secret` for a graceful overlap, then
#    set the new one. In mandible.toml ([jwt] previous_secret = "<old>"), or:
export MANDIBLE__JWT__PREVIOUS_SECRET="$MANDIBLE__JWT__SECRET"
export MANDIBLE__JWT__SECRET="$NEW_JWT_SECRET"

# 3. Restart Mandible
systemctl restart mandible

# 4. After the longest token lifetime has elapsed, drop previous_secret

CA Certificate Rotation

The platform root can be replaced without a flag day. The outgoing root stays trusted and keeps verifying what it already issued while the new one signs, and agents move across as they renew, so nothing has to be re-issued in one pass.

Procedure

# 1. Generate the replacement root into its own directory. Writing it beside the
#    live one would overwrite the root currently signing.
mkdir -p /etc/mantis/certs/new-root
mantisctl cert generate-ca --cn "Mantis Platform Root 2026" --days 1825 \
  --output-dir /etc/mantis/certs/new-root

# 2. Distribute a bundle carrying BOTH roots and restart, before rotating. An
#    agent that renews while the fleet still trusts only the old root receives a
#    certificate nothing accepts.
cat /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/new-root/ca-cert.pem \
  > /etc/mantis/certs/ca-bundle.pem
# install ca-bundle.pem as the trust file on every host, then:
systemctl restart mandible thorax
systemctl restart tarsus@<instance>   # on each agent host

# 3. Hand the replacement to Mandible. The outgoing root is marked retiring in
#    the same transaction: still trusted, no longer signing. Nothing is deleted.
mantisctl cert rotate-root \
  --cert /etc/mantis/certs/new-root/ca-cert.pem \
  --key /etc/mantis/certs/new-root/ca-key.pem

# 4. Watch the fleet move across. Agents renew on their own daily schedule;
#    this reports what still depends on the outgoing root.
mantisctl cert rotation-status

rotation-status counts agents by the authority that actually signed the certificate each one presents. It reports an agent it cannot resolve as unknown rather than as clear, so an unreadable certificate blocks the withdrawal rather than silently permitting it.

Withdrawing the outgoing root

Only once rotation-status reports nothing on the retiring root and nothing unknown. Withdrawing trust while an agent still chains to it takes that agent offline.

mantisctl cert rotation-status   # confirm: safe to withdraw

# Then distribute a trust file containing only the new root and restart.
cp /etc/mantis/certs/new-root/ca-cert.pem /etc/mantis/certs/ca-cert.pem
systemctl restart mandible thorax

Immediate replacement after a compromise

A compromised root is the one case where the overlap is wrong: the point is to stop trusting the old key at once, accepting that every agent must re-enroll. Replace the CA and every component certificate together, restart, and re-enroll the fleet.

mantisctl cert generate-ca --cn "Mantis Platform Root" --days 1825 \
  --output-dir /etc/mantis/certs
mantisctl cert generate-server --name server \
  --sans thorax --sans mandible --sans <hosts...> --output-dir /etc/mantis/certs
mantisctl cert generate-client --name client --output-dir /etc/mantis/certs
mantisctl cert generate-client --name dispatch --cn mandible --output-dir /etc/mantis/certs

systemctl restart tarsus@<instance> thorax mandible

Expect an outage for the length of the re-enrollment.

Post-Rotation Verification

After any rotation, verify the system is fully operational:

# Health checks
curl -s https://localhost:3000/api/v1/health | jq
curl -s https://localhost:3000/api/v1/health/ready
# /health/grpc returns the overall status to anyone; authenticate to also
# receive the breaker internals (circuit_state, failure_count, success_count).
# Without a token those fields come back empty/zeroed.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:3000/api/v1/health/grpc | jq

# Verify encrypted data is accessible. NOTE: `mantisctl key verify` calls the
# running API (needs a JWT with the encryption:verify permission and the services
# restarted first) and is rate-limited to ONE attempt per user per hour -- run it
# exactly once, here, after the restart.
mantisctl key verify

# Test a deployment to confirm end-to-end functionality
# (in a non-production environment)

Next Steps

Certificate Backup — Back up after rotation
Database Backup — Back up after encryption key rotation
Disaster Recovery — Recovery procedures
Encryption at Rest — Encryption architecture