Upgrades

Upgrades

Upgrade Mantis components safely with zero-downtime strategies.

Upgrade Planning

Version Compatibility

Before upgrading, verify compatibility:

Component	Compatibility
Mandible ↔ Thorax	Must match major.minor version
Thorax ↔ Tarsus	Backward compatible within major version
Database schema	Managed by migrations

Pre-Upgrade Checklist

• Review release notes for breaking changes
• Backup database
• Backup configuration files
• Test upgrade in staging environment
• Schedule maintenance window (if needed)
• Notify stakeholders
• Verify rollback plan

Upgrade Order

Always upgrade in this order:

Docker Compose Upgrades

Standard Upgrade

# Images are built locally -- there is no registry to pull from
./docker.sh build          # or: docker compose build

# Stop the app tiers but KEEP Postgres running -- migrations connect to it
docker compose stop mandible thorax lens

# Run migrations against the live database
cd instinct
diesel migration run
cd ..

# Start with the newly built images
docker compose up -d

# Verify health
docker compose ps
curl -k https://localhost:3000/api/v1/health

Rolling Upgrade (Zero Downtime)

Compose cannot scale these services

Both mandible and thorax pin container_name and a fixed host port, so docker compose up --scale is rejected and the ports would collide anyway. Compose is a single-node topology. For genuine multi-instance Thorax use the shipped multi-thorax profile (thorax-2 on port 50053); for real horizontal scaling and zero-downtime rollout use the Kubernetes/Helm deployment.

Rollback

# Rebuild at the previous tag and restart
export MANTIS_IMAGE_TAG=1.2.3
docker compose up -d --build

Kubernetes Upgrades

Mantis publishes no images

Nothing in the repository pushes ghcr.io/mantisplatform/mantis-*; that string is only the default image.registry in deploy/helm/mantis/values.yaml. Build the images yourself (./docker.sh build), push them to a registry your cluster can pull from (./docker.sh push targets $HARBOR_REGISTRY), and set image.registry — or the <registry> placeholder below — to that registry.

Standard Rolling Update

# <registry> is your own; see the caution above.
kubectl set image deployment/mandible mandible=<registry>/mantis-mandible:v1.3.0 -n mantis
kubectl set image deployment/thorax thorax=<registry>/mantis-thorax:v1.3.0 -n mantis
kubectl set image deployment/lens lens=<registry>/mantis-lens:v1.3.0 -n mantis

# Monitor rollout
kubectl rollout status deployment/mandible -n mantis
kubectl rollout status deployment/thorax -n mantis

Blue-Green Deployment

deployments/mandible-v2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mandible-v2
  namespace: mantis
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mantis
      component: mandible
      version: v2
  template:
    metadata:
      labels:
        app: mantis
        component: mandible
        version: v2
    spec:
      containers:
        - name: mandible
          image: <registry>/mantis-mandible:v1.3.0
          # ... rest of spec

Switch traffic:

# Deploy new version
kubectl apply -f deployments/mandible-v2.yaml

# Wait for ready
kubectl rollout status deployment/mandible-v2 -n mantis

# Update service selector to new version
kubectl patch service mandible -n mantis -p '{"spec":{"selector":{"version":"v2"}}}'

# Verify traffic flows to new version
kubectl get endpoints mandible -n mantis

# Remove old deployment after validation
kubectl delete deployment mandible-v1 -n mantis

Canary Deployment

# Deploy canary with subset of traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mandible-canary
  namespace: mantis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mantis
      component: mandible
      track: canary
  template:
    metadata:
      labels:
        app: mantis
        component: mandible
        track: canary
    spec:
      containers:
        - name: mandible
          image: <registry>/mantis-mandible:v1.3.0

Rollback

# Rollback to previous revision
kubectl rollout undo deployment/mandible -n mantis

# Or to specific revision
kubectl rollout undo deployment/mandible --to-revision=2 -n mantis

# Check rollback status
kubectl rollout status deployment/mandible -n mantis

Database Migrations

Running Migrations

Migrations must run before deploying new application code:

# Using diesel CLI (recommended)
cd instinct
diesel migration run

# Kubernetes - the Helm chart runs migrations itself. On an upgrade,
# `<release>-migrate` is a pre-upgrade hook Job (mandible with --migrate-only) that
# Helm waits on before the new pods roll; a first install has no Job, and mandible
# migrates as it boots. The Job is deleted on success, so nothing is left to wait
# on afterwards. Follow it while it runs with:
kubectl logs -n mantis job/<release>-migrate -f

# Set mandible.migrations.enabled=false to run migrations yourself instead.

Migration Best Practices

1. Always backup before migrations

# As a superuser, not `mantis`: audit_log_entries forces row-level security on its
# owner too, so a `mantis` dump silently omits every tenant-scoped audit row -- the
# data this backup exists to protect.
pg_dump -h localhost -U postgres mantis > backup-$(date +%Y%m%d).sql
pg_dumpall -h localhost -U postgres --globals-only > roles-$(date +%Y%m%d).sql

See Database Backup for the full procedure.

2. Run migrations in a transaction

Mantis migrations run in transactions by default. If a migration fails, changes are rolled back.

3. Test migrations on a copy first

# Create test database
createdb mantis_upgrade_test
psql -d mantis_upgrade_test -f backup.sql   # pg_restore only reads archive-format dumps

# Run migration against test database
cd instinct
DATABASE_URL="postgres://localhost/mantis_upgrade_test" diesel migration run

Handling Migration Failures

If a migration fails:

1. Check the error message in logs
2. Do not retry without investigating
3. Fix the underlying issue
4. Migration state is tracked in __diesel_schema_migrations table

-- Check migration status
SELECT * FROM __diesel_schema_migrations ORDER BY run_on DESC;

-- If needed, manually mark migration as rolled back
-- WARNING: Only do this if you understand the consequences
-- The version is the migration directory's timestamp without separators, e.g.
-- 2026-08-31-054756-0000_... is 202608310547560000; it must delete exactly one row.
DELETE FROM __diesel_schema_migrations WHERE version = 'YYYYMMDDHHMMSSNNNN';

Tarsus Agent Upgrades

Upgrade Strategies

Strategy	Description	Downtime
Rolling	Upgrade agents one at a time	None
Batch	Upgrade groups by environment	Minimal
Full	Upgrade all agents simultaneously	Brief

Rolling Upgrade Script

upgrade-agents.sh

#!/bin/bash
# List target IDs via the REST API (there is no `mantisctl targets` subcommand)
AGENTS=$(curl -s -H "Authorization: Bearer $TOKEN" \
    "https://api.mantis.example.com/api/v1/targets" | jq -r '.data[].id')
NEW_VERSION="1.3.0"

for agent in $AGENTS; do
    echo "Upgrading $agent..."

    # Deploy upgrade package with mantisctl
    mantisctl deployment create \
        --solution-id "$TARSUS_UPGRADE_SOLUTION_ID" \
        --target "$agent" \
        --var "version=$NEW_VERSION" \
        --follow

    echo "Completed $agent"
done

Manual Agent Upgrade

On each target server:

# Stop the agent instance (templated unit -- one per agent)
sudo systemctl stop tarsus@web-01

# Backup current binary
sudo cp /usr/bin/tarsus /usr/bin/tarsus.backup

# Download new version
curl -L https://releases.mantis.example.com/tarsus/v1.3.0/tarsus-linux-amd64 \
    -o /usr/bin/tarsus

# Set permissions
sudo chmod 755 /usr/bin/tarsus

# Start the agent instance
sudo systemctl start tarsus@web-01

# Verify
tarsus --version

Configuration Changes

Adding New Settings

New configuration options typically have sensible defaults. For example, Thorax exposes a [session] section and Mandible exposes a [client_session] section:

# Thorax session tuning
[session]
heartbeat_timeout_secs = 90

# Mandible client-session tuning
[client_session]
max_concurrent_commands = 10

Deprecated Settings

Check the release notes for any renamed or removed settings before upgrading. The current database pool keys are max_connections, min_idle, and connection_timeout_secs:

[database]
max_connections = 10
min_idle = 2
connection_timeout_secs = 10

Environment Variable Changes

Update environment files:

# Check for new/changed variables
diff .env.example .env

# Update environment
nano .env

Version-Specific Guides

Breaking Changes Log

Version	Breaking Change	Migration Path
v1.0.0	Initial release	N/A
v1.0.0	The audit log’s entry hash is computed over microsecond timestamps, where it previously used milliseconds.	Every audit row, archive and dead-letter line written before the change hashes differently, so the chain over them cannot be recomputed.

The audit chain does not survive the hash change

On a database carrying pre-change audit rows:

• GET /api/v1/audit/integrity/{tenant_id} reports the chain broken with failure_kind=content_hash, and records that verdict in audit_integrity_checks.
• Retention wedges: archive_and_drop_partition refuses any quarter holding pre-change rows with ChainInvalid, so those partitions are never archived or dropped.
• Archive verification reports every pre-change archive as ChainBroken.
• Startup replay skips pre-change dead-letter lines as corrupt.

Rows written after the upgrade chain normally. There is no rehash tool: the chain exists so that a rewritten row is detectable, and a tool that recomputed it would be one. Verify the integrity of a pre-change chain before upgrading, keep that verdict, and read any post-upgrade failure over pre-change rows against it. A QA or demonstration install that needs a verifiable chain end to end should be created fresh rather than upgraded.

Monitoring Upgrades

Health Checks During Upgrade

# Continuous health monitoring
watch -n 2 'curl -ks https://localhost:3000/api/v1/health | jq'

# Check specific component
curl -k https://localhost:3000/api/v1/health/ready

Key Metrics to Watch

Metric	Normal	Alert
Request latency	< 100ms	> 500ms
Error rate	< 0.1%	> 1%
Active connections	Stable	Spike/drop
Memory usage	Stable	Growing

Log Analysis

# Watch for errors during upgrade
docker compose logs -f --since 5m | grep -i error

# Kubernetes
kubectl logs -l app=mantis -n mantis -f --since=5m | grep -i error

Rollback Procedures

Docker Compose Rollback

# Immediate rollback
docker compose down
export MANTIS_IMAGE_TAG=previous-version
docker compose up -d

# Database rollback (if needed), as a superuser: the audit table's row security
# and roles are not the application user's to restore
psql -U postgres mantis < backup.sql

Kubernetes Rollback

# Rollback deployments
kubectl rollout undo deployment/mandible -n mantis
kubectl rollout undo deployment/thorax -n mantis
kubectl rollout undo deployment/lens -n mantis

# Database rollback (if needed)
kubectl exec -it postgres-0 -n mantis -- psql -U postgres -d mantis -f /backup/backup.sql

Database Rollback

Caution

Database rollbacks may cause data loss. Only perform if absolutely necessary.

# Restore from backup, as a superuser (see Database Backup for why)
pg_restore -h localhost -U postgres -d mantis --clean backup.dump

# Or from SQL dump
psql -h localhost -U postgres -d mantis < backup.sql

Maintenance Windows

Scheduling

For upgrades requiring downtime:

1. Announce maintenance - Notify users 24-48 hours in advance
2. Freeze deployments - Prevent new deployments during upgrade
3. Backup everything - Database, configs, certificates
4. Perform upgrade - Follow upgrade procedures
5. Validate - Run smoke tests
6. Resume operations - Unfreeze and notify users

Creating Deployment Freeze

# In Lens UI: Deployment → Freezes → New Freeze (path /freezes/new)
# Or via API:
curl -X POST https://api.mantis.example.com/api/v1/deployment-freezes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Upgrade Maintenance",
    "start_time": "2024-01-15T02:00:00Z",
    "end_time": "2024-01-15T04:00:00Z"
  }'
# Omitting tenant_id/solution_id/environment_id applies the freeze globally.

Troubleshooting

Upgrade Failed

1. Check logs for specific errors
2. Verify all prerequisites met
3. Ensure database migrations completed
4. Check connectivity between components

Version Mismatch

# Check tarsus version (supports --version via clap)
tarsus --version

# Check mandible and thorax versions via the server info endpoint
curl -ks https://localhost:3000/api/v1/server/info | jq '.server_version'

# Verify the API is healthy after the upgrade
curl -k https://localhost:3000/api/v1/health | jq '.status'

Database Migration Stuck

# Check migration status
psql -U mantis -c "SELECT * FROM __diesel_schema_migrations"

# Check for locks
psql -U mantis -c "SELECT * FROM pg_locks WHERE NOT granted"

Next Steps

• Production Checklist - Verify production readiness
• Backup & Recovery - Backup procedures
• Troubleshooting - Common issues