Key Rotation

Key Rotation

Procedures for rotating cryptographic keys and certificates in Mantis.

Overview

What Needs Rotation

Key Type	Purpose	Recommended Interval
JWT signing key	API authentication	90 days
Encryption master key	Data at rest	365 days
TLS certificates	Transport security	365 days
Agent certificates	mTLS authentication	365 days
CA certificate	Certificate authority	5-10 years

Rotation Triggers

Trigger	Action	Urgency
Scheduled rotation	Plan and execute	Normal
Key compromise	Immediate rotation	Critical
Personnel change	Review and rotate	High
Certificate expiry	Renew before expiry	High
Compliance audit	Rotate as required	Normal

JWT Key Rotation

HS256 Key Rotation

For HMAC-based JWT:

Two mistakes here log every user out

Verification is entirely kid-based: a token carries the key id it was signed with, and a token whose kid does not resolve is rejected. So the incoming key needs a new key_id, and the outgoing one must be moved to previous_key_id. Reusing the same id means the old tokens resolve to the new key and fail.

Setting previous_secret without previous_key_id is worse: the previous key is silently discarded, with no startup error, and every existing token is rejected the moment you restart.

Step-by-step:

1. Generate new key:

NEW_JWT_SECRET=$(openssl rand -base64 32)

2. Set the new key, keeping the old one for verification. Supply these as environment variables, not TOML — MANDIBLE__JWT__* overrides the file, so editing [jwt] while the env var is set leaves the old secret in force and the rotation silently does nothing:

/etc/mantis/mandible.env

MANDIBLE__JWT__SECRET="$NEW_JWT_SECRET"
MANDIBLE__JWT__KEY_ID="k2"            # new id for the new key
MANDIBLE__JWT__PREVIOUS_SECRET="<old secret>"
MANDIBLE__JWT__PREVIOUS_KEY_ID="k1"   # required, or the old key is dropped

3. Restart Mandible:

systemctl restart mandible

4. Wait for old tokens to expire. Access tokens last 1 hour by default; wait at least that long so no live token is still signed with k1.

5. Remove the old key:

MANDIBLE__JWT__SECRET="$NEW_JWT_SECRET"
MANDIBLE__JWT__KEY_ID="k2"
# PREVIOUS_SECRET and PREVIOUS_KEY_ID removed

6. Restart again:

systemctl restart mandible

RS256 Key Rotation

For RSA-based JWT:

1. Generate new key pair:

openssl genrsa -out /etc/mantis/jwt/private-new.pem 2048
openssl rsa -in /etc/mantis/jwt/private-new.pem \
  -pubout -out /etc/mantis/jwt/public-new.pem

2. Point at the new pair and keep the old public key for verification. As with HS256, key_id must change and the outgoing id must move to previous_key_id:

MANDIBLE__JWT__PRIVATE_KEY_PATH="/etc/mantis/jwt/private-new.pem"
MANDIBLE__JWT__PUBLIC_KEY_PATH="/etc/mantis/jwt/public-new.pem"
MANDIBLE__JWT__KEY_ID="k2"                                        # new
MANDIBLE__JWT__PREVIOUS_PUBLIC_KEY_PATH="/etc/mantis/jwt/public-old.pem"
MANDIBLE__JWT__PREVIOUS_KEY_ID="k1"                               # outgoing

Leaving key_id at k1 while also setting previous_key_id = "k1" resolves pre-rotation tokens to the new key, so they all fail verification — the opposite of the overlap this procedure exists to provide.

3. Follow the same restart and cleanup process as HS256

Zero-Downtime JWT Rotation

For high-availability deployments:

Encryption Key Rotation

Master Key Rotation

Rotating the encryption master key requires re-encrypting all data:

Stop the services first, or lose data

Mandible and Thorax read the encryption key once at startup and hold it in memory. If they keep running during rotation, anything they encrypt mid-run is written under the old key — after the rotation pass has already scanned those tables — and becomes permanently undecryptable once they restart on the new key. The failure is silent until someone reads an affected secret. Stop both before rotating.

Step-by-step:

1. Generate new master key:

NEW_MASTER_KEY=$(openssl rand -base64 32)

2. Stop the services that hold the key:

systemctl stop mandible thorax

3. Run the rotation. mantisctl key rotate connects directly to the database and reads the connection string from DATABASE_URL (or the config file) — there is no --database-url flag, so export it first:

export DATABASE_URL="postgres://mantis:...@db-host:5432/mantis"

mantisctl key rotate \
  --old-key "$MANTIS_ENCRYPTION_KEY" \
  --new-key "$NEW_MASTER_KEY" \
  --batch-size 100

4. Monitor progress:

# Key Rotation
#
# Counting encrypted records...
#   storages_s3: 120 records
#   storages_git_auth: 40 records
#   identity_providers: 5 records
#   ...
# Total: 165 records to rotate
#
# Rotating storages_s3...
#   ✓ storages_s3: 120 processed, 120 successful, 0 failed
# Rotating storages_git_auth...
#   ✓ storages_git_auth: 40 processed, 40 successful, 0 failed
# ...
# ✓ 165 records rotated successfully!
# Checkpoint removed.

Rotation covers the platform root and any tenant certificate authority alongside the other encrypted columns, so the signing keys move to the new encryption key with everything else.

5. Set the new key in the services’ environment:

# In the systemd unit env file, not just this shell
MANTIS_ENCRYPTION_KEY="$NEW_MASTER_KEY"

6. Start the services:

systemctl start mandible thorax

7. Verify:

mantisctl key verify --key "$NEW_MASTER_KEY"
#   ✓ storages_s3: 120 records OK
#   ✓ storages_git_auth: 40 records OK
#   ✓ identity_providers: 5 records OK
#   ...
# ✓ 165 records verified successfully!

Partial Rotation Recovery

If rotation fails mid-process, it writes a checkpoint. Resume from where it stopped by re-running the same command with --resume:

mantisctl key rotate \
  --old-key "$MANTIS_ENCRYPTION_KEY" \
  --new-key "$NEW_MASTER_KEY" \
  --resume

TLS Certificate Rotation

Server Certificate Renewal

For Mandible and Thorax server certificates:

1. Generate new certificate:

mantisctl cert generate-server \
  --name mantis --sans mantis.example.com --output-dir /etc/mantis/certs

By hand, the SAN is required — rustls verifies the hostname against it, and a certificate without one is rejected:

openssl req -new -key /etc/mantis/certs/server.key \
  -out /etc/mantis/certs/server-new.csr \
  -subj "/CN=mantis.example.com"

cat > server.ext <<'EOF'
basicConstraints = CA:FALSE
keyUsage         = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName   = DNS:mantis.example.com
EOF

openssl x509 -req -in server-new.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out /etc/mantis/certs/server-new.crt \
  -days 365 -sha256 -extfile server.ext

2. Verify new certificate:

openssl verify -CAfile ca.crt /etc/mantis/certs/server-new.crt

3. Replace certificate:

cp /etc/mantis/certs/server-new.crt /etc/mantis/certs/server.crt

4. Reload service:

# Graceful reload (if supported)
systemctl restart mandible

# Or restart
systemctl restart mandible

Zero-Downtime Certificate Rotation

With load balancer:

rotate-cert.sh

#!/bin/bash
SERVERS="mandible1 mandible2"

for server in $SERVERS; do
  echo "Rotating certificate on $server..."

  # Drain from load balancer
  curl -X POST "http://lb/api/drain/$server"
  sleep 10

  # Update certificate
  scp server-new.crt "$server:/etc/mantis/certs/server.crt"
  ssh "$server" "systemctl restart mandible"

  # Wait for health check
  sleep 5

  # Re-enable in load balancer
  curl -X POST "http://lb/api/enable/$server"

  echo "$server rotated successfully"
done

Agent Certificate Renewal

For Tarsus agent certificates:

Thumbprint mode:

Use tarsus rotate-certificate. It swaps the certificate in place and keeps the agent’s registration and approved status, so the agent never returns to the pending queue:

# On the agent: issue the replacement (both EKUs + SAN, see Certificate Rotation)
mantisctl cert generate-server \
  --name "$(hostname)" \
  --sans "$(hostname -f)" \
  --output-dir /tmp/rotate

tarsus rotate-certificate \
  --new-cert /tmp/rotate/$(hostname)-cert.pem \
  --new-key  /tmp/rotate/$(hostname)-key.pem \
  --reason "scheduled rotation"

systemctl restart tarsus@<instance>

tarsus show-thumbprint can print the wrong thumbprint

Without MANTIS_TLS_CERT_PATH set, show-thumbprint ignores any certificate you just wrote and falls back to the data directory, generating a fresh certificate and printing that one’s thumbprint. Registering it produces an agent that can never connect, because it presents a different certificate.

If you need the thumbprint of a specific file, point at it explicitly:

MANTIS_TLS_CERT_PATH=/etc/mantis/certs/tarsus-cert.pem tarsus show-thumbprint
# or, without touching the agent at all:
mantisctl cert thumbprint /etc/mantis/certs/tarsus-cert.pem

CA-signed mode:

# Generate new CSR
openssl req -new -key /etc/mantis/certs/tarsus-key.pem \
  -out /tmp/agent-new.csr \
  -subj "/CN=$(hostname)"

# Submit to CA for signing
# ... CA signing process ...

# Install new certificate
cp agent-new.crt /etc/mantis/certs/tarsus-cert.pem

# Restart agent
systemctl restart tarsus

CA Certificate Rotation

Planning CA Rotation

CA rotation is complex and requires careful planning:

┌─────────────────────────────────────────────────────────────┐
│  CA Rotation Timeline                                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  T-90 days: Generate new CA                                │
│  T-60 days: Begin issuing certs from new CA               │
│  T-30 days: All new certs use new CA                      │
│  T-0:       Old CA expires                                 │
│                                                             │
│  During overlap:                                            │
│  - Thorax trusts both CAs                                  │
│  - New agents get new CA certs                             │
│  - Old agents continue working                             │
│                                                             │
└─────────────────────────────────────────────────────────────┘

CA Rotation Steps

1. Generate new CA:

openssl genrsa -out /etc/mantis/ca/new-ca.key 4096
openssl req -x509 -new -nodes \
  -key /etc/mantis/ca/new-ca.key \
  -out /etc/mantis/ca/new-ca.crt \
  -days 3650 \
  -subj "/CN=Mantis CA v2"

2. Configure trust for both CAs:

# Concatenate CA certificates
cat /etc/mantis/ca/ca.crt /etc/mantis/ca/new-ca.crt > \
  /etc/mantis/ca/combined-ca.crt

[tls]
ca_cert_path = "/etc/mantis/ca/combined-ca.crt"

3. Replace the stored root. Mandible signs from the certificate authority stored in its database, not from a file, so copying a new certificate over ca.crt changes what Thorax trusts without changing what Mandible signs with. Until the stored root is replaced, every certificate issued still comes from the old one.

   Replacing it is disruptive by design. mantisctl cert import-ca refuses while a root is stored, and a root with tenant certificate authorities beneath it cannot be removed until those are removed first — the database restricts it, so that removing a root cannot silently strand the tenants that chain to it.

   # Tenant authorities first, then the root, then the replacement.
   mantisctl cert import-ca --cert /etc/mantis/ca/new-ca.crt \
                            --key  /etc/mantis/ca/new-ca.key

4. Re-enroll every agent. Certificates issued by the old root keep verifying only while Thorax still trusts it, which is what the combined trust file in step 2 is for. Each agent needs a certificate from the new root before that trust is withdrawn.

5. Withdraw trust in the old CA once no agent presents its certificates:

cp /etc/mantis/ca/new-ca.crt /etc/mantis/ca/ca.crt

Rotating the root is a fleet-wide operation

There is no in-place root rotation: no overlap period in which Mandible signs from two roots, and no automatic re-issue. Plan it as a migration with the trust overlap in step 2 covering the whole re-enrollment, and treat the root’s expiry — which Mandible now warns about 90 days ahead — as the deadline for completing it, not for starting it.

Emergency Rotation

Key Compromise Response

When a key is suspected compromised:

Immediate actions:

1. JWT key compromise:

# Generate new key immediately
NEW_SECRET=$(openssl rand -base64 32)

# Update and restart (accepts service interruption)
sed -i "s/secret = .*/secret = \"$NEW_SECRET\"/" /etc/mantis/mandible.toml
systemctl restart mandible

# All existing tokens invalidated

2. Master encryption key compromise:

# Rotate immediately (may take time for large datasets)
mantisctl key rotate \
  --old-key "$COMPROMISED_KEY" \
  --new-key "$(openssl rand -base64 32)" \
  --force

3. Agent certificate compromise:

# Revoke the compromised registration immediately (by ID or thumbprint prefix)
mantisctl cert revoke "<REGISTRATION_UUID>" \
  --reason "Certificate compromised"

Post-Incident

After emergency rotation:

1. Document the incident
2. Review access logs
3. Identify root cause
4. Update security procedures
5. Conduct security review

Automation

Rotation Reminder Script

check-key-rotation.sh

#!/bin/bash
# Check JWT key age
JWT_KEY_DATE=$(stat -c %Y /etc/mantis/jwt/private.pem 2>/dev/null || echo 0)
NOW=$(date +%s)
AGE_DAYS=$(( (NOW - JWT_KEY_DATE) / 86400 ))

if [ $AGE_DAYS -gt 80 ]; then
  echo "WARNING: JWT key is $AGE_DAYS days old (rotation recommended at 90)"
fi

# Check certificate expiry
CERT_EXPIRY=$(openssl x509 -enddate -noout -in /etc/mantis/certs/server.crt | cut -d= -f2)
EXPIRY_EPOCH=$(date -d "$CERT_EXPIRY" +%s)
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW) / 86400 ))

if [ $DAYS_LEFT -lt 30 ]; then
  echo "WARNING: Server certificate expires in $DAYS_LEFT days"
fi

Automated Certificate Renewal

Using cert-manager (Kubernetes):

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mantis-server
spec:
  secretName: mantis-tls
  duration: 8760h # 1 year
  renewBefore: 720h # 30 days before
  issuerRef:
    name: mantis-ca
    kind: ClusterIssuer
  dnsNames:
    - mantis.example.com

Rotation Schedule

Recommended Schedule

Key/Certificate	Rotation Frequency	Method
JWT signing key	90 days	Overlap rotation
Encryption master key	365 days	Re-encryption
Server TLS certificates	365 days	Reload
Agent certificates	365 days	Per-agent
CA certificate	5-10 years	Planned migration

Calendar Template

┌─────────────────────────────────────────────────────────────┐
│  Annual Rotation Calendar                                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Q1 (Jan-Mar):                                             │
│  - JWT key rotation                                        │
│  - Review certificate expiry dates                         │
│                                                             │
│  Q2 (Apr-Jun):                                             │
│  - JWT key rotation                                        │
│  - Agent certificate batch renewal                         │
│                                                             │
│  Q3 (Jul-Sep):                                             │
│  - JWT key rotation                                        │
│  - Encryption master key rotation                          │
│                                                             │
│  Q4 (Oct-Dec):                                             │
│  - JWT key rotation                                        │
│  - Server certificate renewal                              │
│  - Annual security review                                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Verification

Post-Rotation Checks

After any key rotation:

# Verify JWT authentication
curl -X POST https://mantis.example.com/api/v1/auth/login \
  -d '{"username":"test","password":"test"}' \
  -H "Content-Type: application/json"

# Verify encryption
mantisctl key verify --key "$MANTIS_ENCRYPTION_KEY"

# Verify TLS
# Point at whatever terminates TLS -- usually the reverse proxy on 443.
# Mandible's own port 3000 is plain HTTP unless [server.tls] is configured.
openssl s_client -connect mantis.example.com:443 < /dev/null

# Verify agent connectivity
mantisctl cert list --status approved

Rollback Procedure

If rotation causes issues:

1. JWT keys: Restore previous key to secret and restart
2. Encryption keys: Keep old key, investigate issue
3. Certificates: Restore from backup, restart

# Example: Restore JWT key
cp /backup/jwt-secret /etc/mantis/secrets/jwt-secret
export MANDIBLE__JWT__SECRET=$(cat /etc/mantis/secrets/jwt-secret)
systemctl restart mandible

Best Practices

1. Plan Ahead

• Schedule rotations during low-traffic periods
• Notify teams before rotation
• Have rollback plan ready

2. Test First

• Test rotation in staging environment
• Verify application functionality
• Check monitoring for errors

3. Document Everything

• Record rotation date and time
• Document any issues encountered
• Update runbooks as needed

4. Monitor

• Watch error rates during rotation
• Check authentication success rates
• Monitor agent connectivity

5. Automate Where Possible

• Use cert-manager for certificates
• Implement rotation scripts
• Set up expiry alerts

Next Steps

• JWT Configuration - JWT settings details
• TLS Certificates - Certificate management
• Encryption at Rest - Encryption configuration