Multi-Thorax Deployment

Multi-Thorax Deployment

Deploy multiple Thorax orchestration instances for high availability and horizontal scaling.

Overview

Multi-Thorax deployment enables:

• High availability - Survive instance failures
• Horizontal scaling - Handle more concurrent deployments
• Load distribution - Spread work across instances

Prerequisites

Required Infrastructure

Component	Purpose	Minimum Version
Redis	Cluster coordination	6.0+
RabbitMQ	Command distribution	4.3+
PostgreSQL	Mandible persistence (Thorax reaches it over gRPC, never directly)	16+

Network Requirements

Source	Destination	Port	Protocol
Thorax	Redis	6379	TCP/TLS
Thorax	RabbitMQ	5671	AMQPS
Thorax	Mandible	50052	gRPC/TLS
Tarsus	Mandible	50052	gRPC/mTLS (registration/heartbeat, always)
Tarsus	Thorax	50051	gRPC/mTLS (poll mode only)
Thorax	Tarsus	9342	gRPC/mTLS (listen mode only)

Configuration

Cluster Configuration

Each Thorax instance requires cluster configuration:

thorax.toml

[cluster]
# Enable cluster coordination
enabled = true

# Unique instance identifier
# In Kubernetes, use metadata.name from StatefulSet
instance_id = "thorax-1"

# Redis connection for cluster state
redis_url = "redis://redis:6379"

# Heartbeat frequency (seconds)
heartbeat_interval_secs = 10

# Consider instance dead after this (seconds)
liveness_timeout_secs = 30

# Redis key namespace (allows multi-tenant Redis)
namespace = "mantis-prod"

Environment Variables

# Instance identification
THORAX__CLUSTER__ENABLED=true
THORAX__CLUSTER__INSTANCE_ID=thorax-1
THORAX__CLUSTER__REDIS_URL=redis://redis:6379
THORAX__CLUSTER__NAMESPACE=mantis-prod

# Timing
THORAX__CLUSTER__HEARTBEAT_INTERVAL_SECS=10
THORAX__CLUSTER__LIVENESS_TIMEOUT_SECS=30

Instance Naming

instance_id is the name this instance registers with Mandible under. If it is not configured, it’s auto-generated from the hostname:

Format: thorax-{hostname}
Example: thorax-k8s-worker-2

When instances share a hostname (e.g. containers on the same node), set THORAX__CLUSTER__INSTANCE_ID explicitly: Mandible gives a name to one process at a time. A second process registering a name whose holder is live is refused and keeps retrying; it takes the name once the holder deregisters on a graceful stop or is marked offline when its heartbeat lapses. A holder that comes back after that is refused on its next heartbeat, stops, and exits with an error.

Redis keys use the instance ID Mandible assigns in return, not this name. That is the ID a queued command records as its owner, so the registry can answer whether a command’s owning instance is still alive.

For predictable IDs in Kubernetes, use StatefulSet pod names:

env:
  - name: THORAX__CLUSTER__INSTANCE_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

Redis Cluster State

State Structure

mantis-prod:instances:index          # SET of registered instance IDs
mantis-prod:instances:{id}           # String (JSON) with instance metadata, TTL = liveness_timeout
mantis-prod:affinity:{target_id}     # String (claim info), TTL-based
mantis-prod:affinity:by-instance:{id}  # SET of target IDs owned by an instance
mantis-prod:deployments:{uuid}       # String (JSON) with deployment state, TTL = 1h active / 24h complete
mantis-prod:deployments:active       # SET of UUIDs of running deployments
mantis-prod:deployments:by-instance:{id}  # SET of deployment UUIDs owned by instance

Instance Registration

On startup, each Thorax instance:

1. Registers itself:

   SETEX mantis-prod:instances:01a0b519-67f5-88c1-b926-2e2c1f3a20fd 30 '{"instance_id":"01a0b519-67f5-88c1-b926-2e2c1f3a20fd","started_at":1705312200,...}'
   SADD mantis-prod:instances:index 01a0b519-67f5-88c1-b926-2e2c1f3a20fd

2. Sends heartbeats (every 10s):

   The heartbeat updates the last_heartbeat timestamp inside the instance JSON and rewrites the key with a refreshed TTL — there is no separate heartbeat key. Check liveness with:

   TTL mantis-prod:instances:01a0b519-67f5-88c1-b926-2e2c1f3a20fd

3. Loads existing sessions from Mandible

Instance Cleanup

When an instance stops sending heartbeats:

1. Other instances detect missing heartbeat (TTL expired)
2. Dead instance removed from instances SET
3. Its target affinity claims are deleted, so the targets are free — nothing claims them on its behalf; the next instance to dispatch to a target takes it
4. In-flight deployments on that instance are marked failed, not retried

Deployment Process

Creating Instances

Docker Compose:

services:
  thorax-1:
    image: ghcr.io/mantisplatform/mantis-thorax:1.0.0
    environment:
      - THORAX__CLUSTER__ENABLED=true
      - THORAX__CLUSTER__INSTANCE_ID=thorax-1
      - THORAX__CLUSTER__REDIS_URL=redis://redis:6379
    volumes:
      - ./certs:/certs:ro

  thorax-2:
    image: ghcr.io/mantisplatform/mantis-thorax:1.0.0
    environment:
      - THORAX__CLUSTER__ENABLED=true
      - THORAX__CLUSTER__INSTANCE_ID=thorax-2
      - THORAX__CLUSTER__REDIS_URL=redis://redis:6379
    volumes:
      - ./certs:/certs:ro

Kubernetes StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thorax
spec:
  replicas: 3
  serviceName: thorax
  selector:
    matchLabels:
      app: thorax
  template:
    metadata:
      labels:
        app: thorax
    spec:
      containers:
        - name: thorax
          image: ghcr.io/mantisplatform/mantis-thorax:1.0.0
          env:
            - name: THORAX__CLUSTER__ENABLED
              value: 'true'
            - name: THORAX__CLUSTER__INSTANCE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: THORAX__CLUSTER__REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: thorax-secrets
                  key: redis-url
          ports:
            - containerPort: 50051
              name: grpc
          volumeMounts:
            - name: certs
              mountPath: /certs
              readOnly: true

Headless Service for Target Connections

Targets need direct connections to specific Thorax instances:

apiVersion: v1
kind: Service
metadata:
  name: thorax
spec:
  clusterIP: None # Headless
  selector:
    app: thorax
  ports:
    - port: 50051
      name: grpc

Each pod is addressable as:

• thorax-0.thorax.namespace.svc.cluster.local
• thorax-1.thorax.namespace.svc.cluster.local
• thorax-2.thorax.namespace.svc.cluster.local

Scaling Operations

Scale Up

# Docker Compose. Do NOT use `--scale thorax=3`: the shipped thorax service pins
# container_name and a fixed host port, so scaling collides. Use the repo's
# multi-thorax profile instead (extra thorax replicas on :50053):
docker compose --profile multi-thorax up -d

# Kubernetes - increase replicas
kubectl scale statefulset thorax --replicas=5

New instances automatically:

1. Register in Redis
2. Start consuming from RabbitMQ
3. Accept target connections
4. Participate in deployment work

Scale Down

# Kubernetes - decrease replicas
kubectl scale statefulset thorax --replicas=2

Scale-down process:

1. Instance receives SIGTERM
2. Stops accepting new work
3. Completes in-flight deployments (graceful period)
4. Deregisters from Redis
5. Target affinity claims are released, and another instance claims each target on its next dispatch

Rolling Updates

# Kubernetes - rolling restart
kubectl rollout restart statefulset thorax

# Watch progress
kubectl rollout status statefulset thorax

Monitoring

Instance Health

# Check active instances
redis-cli SMEMBERS mantis-prod:instances:index

# Check instance details (stored as a JSON string)
redis-cli GET mantis-prod:instances:01a0b519-67f5-88c1-b926-2e2c1f3a20fd

# Check instance liveness (TTL on the instance key; 0 or -2 means expired/gone)
redis-cli TTL mantis-prod:instances:01a0b519-67f5-88c1-b926-2e2c1f3a20fd

Metrics

Thorax exposes its Prometheus metrics at :9090/metrics, all under the mantis_ prefix. There are no dedicated thorax_cluster_* or thorax_heartbeat_latency_ms series; use the per-instance series below (scrape each instance as a separate target and aggregate):

Metric	Type	Description
mantis_deployments_active	Gauge	Active deployments on the instance
mantis_commands_dispatched_total	Counter	Commands dispatched by the instance

Alerts

# Prometheus AlertManager rules
groups:
  - name: thorax-cluster
    rules:
      - alert: ThoraxInstanceDown
        # 'up' is the Prometheus scrape-health series for the thorax job
        expr: count(up{job="mantis-thorax"} == 1) < 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Fewer than 2 Thorax instances are being scraped successfully'

      - alert: ThoraxNoActiveDeployments
        expr: sum(mantis_deployments_active) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'No active deployments across the Thorax cluster'

Failover Behavior

Instance Failure

A Thorax holds the plans of the deployments it runs in memory, so a deployment an instance was running when it stopped does not survive it. Mandible marks it failed, naming the restart, when the instance registers again or, if it stays down, once its heartbeat has lapsed. After a graceful stop, which deregisters it, the instance registers again at once. After a crash, or a stop that could not reach Mandible, it waits until it is marked offline, 60 to 90 seconds at the defaults and longer under a heartbeat override, because until then the name is held by the process that stopped. Redeploy it once the instance is back.

Target Reconnection

When an instance fails, targets:

1. Detect connection loss
2. Re-register with Mandible to obtain a new Thorax assignment
3. Reconnect to the newly assigned instance
4. Resume operations

Default reconnection timing (exponential backoff with jitter):

• Initial retry delay: 1s (retry.initial_delay_ms)
• Backoff multiplier: ×2 per attempt (retry.backoff_multiplier)
• Maximum delay cap: 60s (retry.max_delay_ms)
• Jitter: ±10% of the computed delay
• Maximum retries: Unlimited (retry.retry_indefinitely defaults to true)

Troubleshooting

Instance Not Joining Cluster

1. Check Redis connectivity:

   redis-cli -h $REDIS_HOST ping

2. Verify configuration:

   # Check instance ID is unique
   redis-cli SMEMBERS mantis-prod:instances:index

3. Check logs:

   docker logs thorax-1 | grep -i cluster

Targets Not Reconnecting

1. Check target affinity:

   redis-cli GET mantis-prod:affinity:target_id

2. Verify instance is healthy:

   redis-cli TTL mantis-prod:instances:01a0b519-67f5-88c1-b926-2e2c1f3a20fd

3. Force rebalance (if needed):

   redis-cli DEL mantis-prod:affinity:target_id

Split-Brain Concerns

Redis holds the shared state, but it does not fence a partitioned instance:

• Instance registration is atomic and heartbeats use a TTL for automatic cleanup
• An affinity claim carries no fencing token — the stored value is {instance_id}: and both the release path and the ownership check compare that prefix and nothing else
• The affinity check fails open: an instance that cannot reach Redis after its retries proceeds with the dispatch anyway

Mandible does fence the instance name: every registration carries a per-boot ID, and a boot that no longer holds the name is refused at its next heartbeat and stops. Until that heartbeat, up to one interval, a holder returning from a partition can still dispatch and report under the instance’s identity, since the other internal calls carry no boot ID.

See Cluster Coordination for the full model.

Best Practices

Production Deployment

1. Minimum 2 instances for HA
2. 3+ instances for production workloads
3. Spread across zones for fault tolerance

Configuration

1. Use StatefulSet for stable network identities
2. Configure PodDisruptionBudget for safe updates
3. Set resource limits to prevent noisy neighbors
4. Enable Redis persistence for state durability

Operations

1. Monitor instance liveness - watch up{job="mantis-thorax"}, the scrape-health series the ThoraxInstanceDown rule above alerts on, alongside the TTL on each instance’s Redis registry key; Thorax publishes no session-count or heartbeat-latency series
2. Test failover regularly - Kill instances intentionally
3. Document recovery procedures - Runbooks for incidents
4. Automate scaling - HPA based on queue depth

Next Steps

• Load Balancing - Route traffic to instances
• Competing Consumers - Mandible replicas draining the dispatch queue
• Instance Affinity - Target binding strategies