Key Rotation

Key Rotation

Procedures for rotating cryptographic keys and certificates in Mantis.

Overview

What Needs Rotation

Key Type	Purpose	Recommended Interval
JWT signing key	API authentication	90 days
Encryption master key	Data at rest	365 days
TLS certificates	Transport security	365 days
Agent certificates	mTLS authentication	365 days
CA certificate	Certificate authority	5-10 years

Rotation Triggers

Trigger	Action	Urgency
Scheduled rotation	Plan and execute	Normal
Key compromise	Immediate rotation	Critical
Personnel change	Review and rotate	High
Certificate expiry	Renew before expiry	High
Compliance audit	Rotate as required	Normal

JWT Key Rotation

HS256 Key Rotation

For HMAC-based JWT:

Two mistakes here log every user out

Verification is entirely kid-based: a token carries the key id it was signed with, and a token whose kid does not resolve is rejected. So the incoming key needs a new key_id, and the outgoing one must be moved to previous_key_id. Reusing the same id means the old tokens resolve to the new key and fail.

Setting previous_secret without previous_key_id — or either one without the other — is refused at startup, as is a previous_key_id equal to key_id. Both configurations produce a single usable key, which would reject every existing token on restart; Mandible declines to start rather than do that quietly.

Step-by-step:

1. Generate new key:

NEW_JWT_SECRET=$(openssl rand -base64 32)

2. Set the new key, keeping the old one for verification. Supply these as environment variables, not TOML — MANDIBLE__JWT__* overrides the file, so editing [jwt] while the env var is set leaves the old secret in force and the rotation silently does nothing:

/etc/mantis/mandible.env

MANDIBLE__JWT__SECRET="$NEW_JWT_SECRET"
MANDIBLE__JWT__KEY_ID="k2"            # new id for the new key
MANDIBLE__JWT__PREVIOUS_SECRET="<old secret>"
MANDIBLE__JWT__PREVIOUS_KEY_ID="k1"   # required, or the old key is dropped

3. Restart Mandible:

systemctl restart mandible

4. Wait for old tokens to expire. Access tokens last 1 hour by default; wait at least that long so no live token is still signed with k1.

5. Remove the old key:

MANDIBLE__JWT__SECRET="$NEW_JWT_SECRET"
MANDIBLE__JWT__KEY_ID="k2"
# PREVIOUS_SECRET and PREVIOUS_KEY_ID removed

6. Restart again:

systemctl restart mandible

RS256 Key Rotation

For RSA-based JWT:

1. Generate new key pair:

openssl genrsa -out /etc/mantis/jwt/private-new.pem 2048
openssl rsa -in /etc/mantis/jwt/private-new.pem \
  -pubout -out /etc/mantis/jwt/public-new.pem

2. Point at the new pair and keep the old public key for verification. As with HS256, key_id must change and the outgoing id must move to previous_key_id:

MANDIBLE__JWT__PRIVATE_KEY_PATH="/etc/mantis/jwt/private-new.pem"
MANDIBLE__JWT__PUBLIC_KEY_PATH="/etc/mantis/jwt/public-new.pem"
MANDIBLE__JWT__KEY_ID="k2"                                        # new
MANDIBLE__JWT__PREVIOUS_PUBLIC_KEY_PATH="/etc/mantis/jwt/public-old.pem"
MANDIBLE__JWT__PREVIOUS_KEY_ID="k1"                               # outgoing

Leaving key_id at k1 while also setting previous_key_id = "k1" resolves pre-rotation tokens to the new key, so they all fail verification — the opposite of the overlap this procedure exists to provide.

3. Follow the same restart and cleanup process as HS256

Zero-Downtime JWT Rotation

For high-availability deployments:

Encryption Key Rotation

Master Key Rotation

Rotating the encryption master key requires re-encrypting all data:

Stop Mandible first, or lose data

Mandible reads the encryption key once at startup and holds it in memory. If it keeps running during rotation, anything it encrypts mid-run is written under the old key — after the rotation pass has already scanned those tables — and becomes permanently undecryptable once it restarts on the new key. The failure is silent until someone reads an affected secret. Stop Mandible before rotating.

Thorax needs no such stop: it has no database connection, no [encryption] config section and no encryption key. Mandible does the decrypting and hands Thorax the resolved action payload over the internal gRPC channel.

Step-by-step:

1. Generate new master key:

NEW_MASTER_KEY=$(openssl rand -base64 32)

2. Stop Mandible, the only service that holds the key:

systemctl stop mandible

3. Run the rotation. mantisctl key rotate connects directly to the database and reads the connection string from DATABASE_URL (or the config file) — there is no --database-url flag, so export it first:

export DATABASE_URL="postgres://mantis:...@db-host:5432/mantis"

# Both keys come from the environment; passed as arguments they would be visible
# in the process list for the whole rotation. `--old-key` is required, so omitting
# MANTIS_ENCRYPTION_KEY aborts the command.
export MANTIS_ENCRYPTION_KEY="$OLD_MASTER_KEY"
export MANTIS_NEW_ENCRYPTION_KEY="$NEW_MASTER_KEY"

mantisctl key rotate --batch-size 100

4. Monitor progress:

# Key Rotation
#
# Counting records to scan...
#   command_queue: 0 records
#   storages_s3: 120 records
#   storages_git_auth: 40 records
#   identity_providers: 5 records
#   ...
# Total: 165 records to scan
#
# Rotating command_queue...
#   ✓ command_queue: 0 processed, 0 successful, 0 failed
# Rotating storages_s3...
#   ✓ storages_s3: 120 processed, 120 successful, 0 failed
# Rotating storages_git_auth...
#   ✓ storages_git_auth: 40 processed, 40 successful, 0 failed
# ...
# ✓ 165 records rotated successfully!
# Checkpoint removed.

Rotation covers the platform root and any tenant certificate authority alongside the other encrypted columns, so the signing keys move to the new encryption key with everything else.

5. Set the new key in Mandible’s environment:

# In the systemd unit env file, not just this shell
MANTIS_ENCRYPTION_KEY="$NEW_MASTER_KEY"

6. Start Mandible:

systemctl start mandible

7. Verify:

mantisctl key verify --key "$NEW_MASTER_KEY"
#   ✓ command_queue: 0 records OK
#   ✓ storage_credentials: 120 records OK
#   ✓ git_credentials: 40 records OK
#   ✓ identity_providers: 5 records OK
#   ...
# ✓ 165 records verified successfully!

Partial Rotation Recovery

If rotation fails mid-process, it writes a checkpoint. Resume from where it stopped by re-running the same command with --resume:

mantisctl key rotate --resume

TLS Certificate Rotation

Server Certificate Renewal

For Mandible and Thorax server certificates:

1. Generate new certificate:

mantisctl cert generate-server \
  --name mantis --sans mantis.example.com --output-dir /etc/mantis/certs \
  --ca-cert /etc/mantis/ca/ca.crt --ca-key /etc/mantis/ca/ca.key

--ca-cert and --ca-key default to ./certs/ca-cert.pem and ./certs/ca-key.pem, so pass them explicitly unless you are running from a directory laid out that way.

By hand, the SAN is required — rustls verifies the hostname against it, and a certificate without one is rejected:

openssl req -new -key /etc/mantis/certs/server.key \
  -out /etc/mantis/certs/server-new.csr \
  -subj "/CN=mantis.example.com"

cat > server.ext <<'EOF'
basicConstraints = CA:FALSE
keyUsage         = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName   = DNS:mantis.example.com
EOF

openssl x509 -req -in server-new.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out /etc/mantis/certs/server-new.crt \
  -days 365 -sha256 -extfile server.ext

2. Verify new certificate:

openssl verify -CAfile ca.crt /etc/mantis/certs/server-new.crt

3. Replace certificate:

cp /etc/mantis/certs/server-new.crt /etc/mantis/certs/server.crt

4. Restart the service (certificates are read at startup; there is no reload):

systemctl restart mandible

Zero-Downtime Certificate Rotation

With load balancer:

rotate-cert.sh

#!/bin/bash
SERVERS="mandible1 mandible2"

for server in $SERVERS; do
  echo "Rotating certificate on $server..."

  # Drain from load balancer
  curl -X POST "http://lb/api/drain/$server"
  sleep 10

  # Update certificate
  scp server-new.crt "$server:/etc/mantis/certs/server.crt"
  ssh "$server" "systemctl restart mandible"

  # Wait for health check
  sleep 5

  # Re-enable in load balancer
  curl -X POST "http://lb/api/enable/$server"

  echo "$server rotated successfully"
done

Agent Certificate Renewal

For Tarsus agent certificates:

Thumbprint mode:

Use tarsus rotate-certificate. It swaps the certificate in place and keeps the agent’s registration and approved status, so the agent never returns to the pending queue. Before it asks, it connects to Mandible with the new pair and refuses the rotation if the server will not accept it: the server retires the old certificate when it commits, so a replacement from the wrong CA would otherwise leave the agent with no identity it can use.

# On the agent: issue the replacement (both EKUs + SAN, see Certificate Rotation)
mantisctl cert generate-server \
  --name "$(hostname)" \
  --sans "$(hostname -f)" \
  --output-dir /tmp/rotate \
  --ca-cert /etc/mantis/ca/ca.crt \
  --ca-key /etc/mantis/ca/ca.key

tarsus rotate-certificate \
  --new-cert /tmp/rotate/$(hostname)-cert.pem \
  --new-key  /tmp/rotate/$(hostname)-key.pem \
  --reason "scheduled rotation"

systemctl restart tarsus@<instance>

tarsus show-thumbprint can print the wrong thumbprint

Without MANTIS_TLS_CERT_PATH set, show-thumbprint ignores any certificate you just wrote and falls back to the data directory, generating a fresh certificate and printing that one’s thumbprint. Registering it produces an agent that can never connect, because it presents a different certificate.

If you need the thumbprint of a specific file, point at it explicitly:

MANTIS_TLS_CERT_PATH=/etc/mantis/certs/tarsus-cert.pem tarsus show-thumbprint
# or, without touching the agent at all:
mantisctl cert thumbprint /etc/mantis/certs/tarsus-cert.pem

CA-signed mode:

# Generate new CSR
openssl req -new -key /etc/mantis/certs/tarsus-key.pem \
  -out /tmp/agent-new.csr \
  -subj "/CN=$(hostname)"

# Submit to CA for signing
# ... CA signing process ...

# Install new certificate
cp agent-new.crt /etc/mantis/certs/tarsus-cert.pem

# Restart agent
systemctl restart tarsus

CA Certificate Rotation

Planning CA Rotation

CA rotation is complex and requires careful planning:

┌─────────────────────────────────────────────────────────────┐
│  CA Rotation Timeline                                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  T-90 days: Generate new CA                                │
│  T-60 days: Begin issuing certs from new CA               │
│  T-30 days: All new certs use new CA                      │
│  T-0:       Old CA expires                                 │
│                                                             │
│  During overlap:                                            │
│  - Thorax trusts both CAs                                  │
│  - New agents get new CA certs                             │
│  - Old agents continue working                             │
│                                                             │
└─────────────────────────────────────────────────────────────┘

CA Rotation Steps

1. Generate new CA:

openssl genrsa -out /etc/mantis/ca/new-ca.key 4096
openssl req -x509 -new -nodes \
  -key /etc/mantis/ca/new-ca.key \
  -out /etc/mantis/ca/new-ca.crt \
  -days 3650 \
  -subj "/CN=Mantis CA v2"

2. Configure trust for both CAs:

# Concatenate CA certificates
cat /etc/mantis/ca/ca.crt /etc/mantis/ca/new-ca.crt > \
  /etc/mantis/ca/combined-ca.crt

[tls]
ca_cert_path = "/etc/mantis/ca/combined-ca.crt"

3. Replace the stored root. Mandible signs from the certificate authority stored in its database, not from a file, so copying a new certificate over ca.crt changes what Thorax trusts without changing what Mandible signs with. cert rotate-root replaces it in place: the new authority signs from now on and the outgoing one is marked retiring in the same transaction — still trusted, no longer signing. Nothing is deleted, and no tenant authority has to be removed first.

   The combined trust file from step 2 must already be in place on every host: an agent that renews while the fleet still trusts only the old root presents a certificate nothing accepts.

   mantisctl cert rotate-root --cert /etc/mantis/ca/new-ca.crt \
                              --key  /etc/mantis/ca/new-ca.key

   mantisctl cert import-ca is for the first root only — it refuses with CertificateAlreadyExists while one is stored.

4. Re-enroll every agent. Certificates issued by the old root keep verifying only while Thorax still trusts it, which is what the combined trust file in step 2 is for. Each agent needs a certificate from the new root before that trust is withdrawn.

5. Withdraw trust in the old CA once nothing still chains to it:

# Gate on this: it counts agents by the authority that signed the certificate
# they present, and reports an unresolvable one as unknown rather than as clear.
mantisctl cert rotation-status
# Marks the retiring root retired, after which Mantis trusts nothing it signed.
# Refused while anything still chains to it.
mantisctl cert withdraw-root

cp /etc/mantis/ca/new-ca.crt /etc/mantis/ca/ca.crt

Rotating the root is a fleet-wide operation

rotate-root gives the overlap — both roots trusted, one signing — but not the re-issue: every agent still needs a new certificate before the old root’s trust is withdrawn. Plan it as a migration with the trust overlap in step 2 covering the whole re-enrollment, and treat the root’s expiry — which Mandible warns about 90 days ahead — as the deadline for completing it, not for starting it.

Emergency Rotation

Key Compromise Response

When a key is suspected compromised:

Immediate actions:

1. JWT key compromise:

# Generate new key immediately
NEW_SECRET=$(openssl rand -base64 32)

# Set it where the scheduled procedure sets it: MANDIBLE__JWT__* overrides the
# TOML, so editing [jwt] while the env var is set leaves the compromised key live.
sed -i "s|^MANDIBLE__JWT__SECRET=.*|MANDIBLE__JWT__SECRET=\"$NEW_SECRET\"|" \
  /etc/mantis/mandible.env
systemctl restart mandible

# Every token signed with the old key now fails to verify. Drop
# MANDIBLE__JWT__PREVIOUS_SECRET/PREVIOUS_KEY_ID too, or the compromised key
# keeps verifying.

2. Master encryption key compromise:

# Rotate immediately (may take time for large datasets)
export MANTIS_ENCRYPTION_KEY="$COMPROMISED_KEY"
export MANTIS_NEW_ENCRYPTION_KEY="$(openssl rand -base64 32)"
mantisctl key rotate --force

3. Agent certificate compromise:

# Revoke the compromised registration immediately (by ID or thumbprint prefix)
mantisctl cert revoke "<REGISTRATION_UUID>" \
  --reason "Certificate compromised"

Post-Incident

After emergency rotation:

1. Document the incident
2. Review access logs
3. Identify root cause
4. Update security procedures
5. Conduct security review

Automation

Rotation Reminder Script

check-key-rotation.sh

#!/bin/bash
# Check JWT key age
JWT_KEY_DATE=$(stat -c %Y /etc/mantis/jwt/private.pem 2>/dev/null || echo 0)
NOW=$(date +%s)
AGE_DAYS=$(( (NOW - JWT_KEY_DATE) / 86400 ))

if [ $AGE_DAYS -gt 80 ]; then
  echo "WARNING: JWT key is $AGE_DAYS days old (rotation recommended at 90)"
fi

# Check certificate expiry
CERT_EXPIRY=$(openssl x509 -enddate -noout -in /etc/mantis/certs/server.crt | cut -d= -f2)
EXPIRY_EPOCH=$(date -d "$CERT_EXPIRY" +%s)
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW) / 86400 ))

if [ $DAYS_LEFT -lt 30 ]; then
  echo "WARNING: Server certificate expires in $DAYS_LEFT days"
fi

Automated Certificate Renewal

Using cert-manager (Kubernetes):

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mantis-server
spec:
  secretName: mantis-tls
  duration: 8760h # 1 year
  renewBefore: 720h # 30 days before
  issuerRef:
    name: mantis-ca
    kind: ClusterIssuer
  dnsNames:
    - mantis.example.com

Rotation Schedule

Recommended Schedule

Key/Certificate	Rotation Frequency	Method
JWT signing key	90 days	Overlap rotation
Encryption master key	365 days	Re-encryption
Server TLS certificates	365 days	Restart
Agent certificates	365 days	Per-agent
CA certificate	5-10 years	Planned migration

Calendar Template

┌─────────────────────────────────────────────────────────────┐
│  Annual Rotation Calendar                                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Q1 (Jan-Mar):                                             │
│  - JWT key rotation                                        │
│  - Review certificate expiry dates                         │
│                                                             │
│  Q2 (Apr-Jun):                                             │
│  - JWT key rotation                                        │
│  - Agent certificate batch renewal                         │
│                                                             │
│  Q3 (Jul-Sep):                                             │
│  - JWT key rotation                                        │
│  - Encryption master key rotation                          │
│                                                             │
│  Q4 (Oct-Dec):                                             │
│  - JWT key rotation                                        │
│  - Server certificate renewal                              │
│  - Annual security review                                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Verification

Post-Rotation Checks

After any key rotation:

# Verify JWT authentication
curl -X POST https://mantis.example.com/api/v1/auth/login \
  -d '{"email":"user@example.com","password":"your-password"}' \
  -H "Content-Type: application/json"

# Verify encryption
mantisctl key verify --key "$MANTIS_ENCRYPTION_KEY"

# Verify TLS
# Point at whatever terminates TLS -- usually the reverse proxy on 443.
# Mandible's own port 3000 is plain HTTP unless [server.tls] is configured.
openssl s_client -connect mantis.example.com:443 < /dev/null

# Verify agent connectivity
mantisctl cert list --status approved

Rollback Procedure

If rotation causes issues:

1. JWT keys: Restore previous key to secret and restart
2. Encryption keys: Keep old key, investigate issue
3. Certificates: Restore from backup, restart

# Example: Restore JWT key
cp /backup/jwt-secret /etc/mantis/secrets/jwt-secret
export MANDIBLE__JWT__SECRET=$(cat /etc/mantis/secrets/jwt-secret)
systemctl restart mandible

Best Practices

1. Plan Ahead

• Schedule rotations during low-traffic periods
• Notify teams before rotation
• Have rollback plan ready

2. Test First

• Test rotation in staging environment
• Verify application functionality
• Check monitoring for errors

3. Document Everything

• Record rotation date and time
• Document any issues encountered
• Update runbooks as needed

4. Monitor

• Watch error rates during rotation
• Check authentication success rates
• Monitor agent connectivity

5. Automate Where Possible

• Use cert-manager for certificates
• Implement rotation scripts
• Set up expiry alerts

Next Steps

• JWT Configuration - JWT settings details
• TLS Certificates - Certificate management
• Encryption at Rest - Encryption configuration