Disaster Recovery

Disaster Recovery

This guide covers recovering a Mantis deployment from catastrophic failure, including complete data loss, infrastructure failure, and key compromise scenarios.

Recovery Requirements

To fully recover a Mantis deployment, you need three things:

Component	Source	Why It Is Needed
Database backup	pg_dump or cloud snapshot	All configuration, deployment history, audit logs
Encryption master key	Secrets manager or offline backup	Decrypt credentials, secrets, and sensitive variables
TLS certificates and CA key	Certificate backup	Re-establish mTLS trust between components

Danger

If any of these three are permanently lost, some data will be unrecoverable. Test your backup procedures regularly.

Recovery Scenarios

Scenario 1: Single Component Failure

Impact: One Mantis component (Mandible, Thorax, or Tarsus) is lost.

Recovery:

1. Re-install the component on the same or new host
2. Restore the component’s TLS certificate and key from backup
3. Update configuration to point to existing infrastructure (database, RabbitMQ)
4. Start the component

# Example: Restore and restart Mandible
cp /opt/mantis/backups/certs/mandible.* /etc/mantis/certs/
cp /opt/mantis/backups/config/mandible.toml /etc/mantis/

# Set the encryption key
export MANTIS_ENCRYPTION_KEY="$(vault kv get -field=key secret/mantis/encryption)"

systemctl start mandible

# Verify health
curl -s https://localhost:3000/api/v1/health | jq

The database and other components are unaffected. Thorax and Tarsus will reconnect automatically.

Scenario 2: Database Loss

Impact: PostgreSQL data is lost. All configuration, deployment history, and audit logs are gone.

Recovery:

1. Provision a new PostgreSQL instance
2. Restore from the latest database backup
3. Verify the encryption master key matches the backup
4. Restart all Mantis components

# Provision new database
createdb -h new-db-host -U postgres mantis

# Restore from backup
pg_restore -h new-db-host -U mantis -d mantis \
  /opt/mantis/backups/db/mantis-latest.dump

# Update Mandible configuration
export MANDIBLE__DATABASE__URL="postgres://mantis:password@new-db-host:5432/mantis"

# Restart services
systemctl restart mandible

Data loss window: From the time of the last backup to the time of failure. Consider enabling PostgreSQL WAL archiving for point-in-time recovery to minimize this window.

Scenario 3: Complete Infrastructure Loss

Impact: All servers, databases, and services are lost (data center failure).

Recovery procedure:

1. Provision infrastructure — Set up new servers, PostgreSQL, RabbitMQ, and Redis (Redis is required — JWT revocation fails closed and SSE is non-functional without it)
2. Restore certificates — Extract from encrypted off-site backup
3. Restore database — Restore from off-site backup or cloud snapshot
4. Deploy components — Install Mandible, Thorax, and Tarsus
5. Configure and start — Apply configuration with correct endpoints
6. Re-register agents — Tarsus agents may need to re-register

# Step 1: Infrastructure
# (Provision via Terraform, CloudFormation, or manually)

# Step 2: Restore certificates
gpg --decrypt /offsite/certs-backup.tar.gz.gpg | tar xzf - -C /etc/mantis/

# Step 3: Restore database
pg_restore -h localhost -U mantis -d mantis /offsite/mantis-latest.dump

# Step 4-5: Deploy and configure
# (Use your standard deployment procedure)

# Step 6: Verify
curl -s https://new-mandible:3000/api/v1/health | jq
# /health/grpc returns the overall status to any caller; pass a token to also
# see the breaker internals (circuit_state, failure_count), which anonymous
# requests do not receive.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://new-mandible:3000/api/v1/health/grpc | jq

Scenario 4: Encryption Key Compromise

Impact: The encryption master key has been exposed to an unauthorized party.

Immediate actions:

1. Rotate the master key immediately
2. Re-encrypt all data with the new key
3. Rotate all credentials that may have been decrypted
4. Review audit logs for unauthorized access

# Generate new master key
NEW_KEY=$(openssl rand -base64 32)

# Rotate encryption key (re-encrypts all data)
mantisctl key rotate --old-key "$OLD_KEY" --new-key "$NEW_KEY"

# Update the key in your secrets manager
vault kv put secret/mantis/encryption key="$NEW_KEY"

# Restart services with new key
export MANTIS_ENCRYPTION_KEY="$NEW_KEY"
systemctl restart mandible thorax

Scenario 5: CA Key Compromise

Impact: An attacker can sign certificates for any component.

Immediate actions:

1. Generate a new CA key pair
2. Re-sign all component certificates with the new CA
3. Distribute new certificates to all components
4. Revoke the old CA certificate
5. Restart all services

This is the most disruptive recovery scenario. Plan and practice this procedure in advance.

Recovery Time Objectives

Scenario	Expected RTO	Depends On
Single component failure	15-30 minutes	Backup availability, deployment automation
Database loss	30-60 minutes	Backup size, restore speed
Complete infrastructure loss	2-4 hours	Infrastructure provisioning, backup retrieval
Key compromise	1-2 hours	Re-encryption time, credential rotation scope

Recovery Testing

Test your recovery procedures regularly:

Monthly: Backup Verification

# Restore database to temporary instance and verify
createdb mantis_dr_test
pg_restore -d mantis_dr_test /opt/mantis/backups/db/mantis-latest.dump
psql -d mantis_dr_test -c "SELECT count(*) FROM tenants"
dropdb mantis_dr_test

Quarterly: Full Recovery Drill

1. Provision isolated test infrastructure
2. Restore all backups (database, certificates, configuration)
3. Start all components
4. Verify deployments can be created and executed
5. Document any issues encountered
6. Update procedures based on findings

Prevention Checklist

• Daily database backups with off-site copies
• Weekly certificate backup verification
• Encryption master key stored in secrets manager with backup
• CA key stored offline or in HSM
• WAL archiving enabled for point-in-time recovery
• Recovery procedures documented and tested quarterly
• Alert on backup job failures
• Backup retention meets compliance requirements

Next Steps

• Database Backup — Database backup procedures
• Certificate Backup — Certificate backup procedures
• Key Rotation Runbook — Scheduled rotation procedures
• Encryption at Rest — Master key management