Multi-Thorax Deployment

Multi-Thorax Deployment

Deploy multiple Thorax orchestration instances for high availability and horizontal scaling.

Overview

Multi-Thorax deployment enables:

• High availability - Survive instance failures
• Horizontal scaling - Handle more concurrent deployments
• Load distribution - Spread work across instances

Prerequisites

Required Infrastructure

Component	Purpose	Minimum Version
Redis	Cluster coordination	6.0+
RabbitMQ	Command distribution	3.8+
PostgreSQL	Shared state	16+

Network Requirements

Source	Destination	Port	Protocol
Thorax	Redis	6379	TCP/TLS
Thorax	RabbitMQ	5671	AMQPS
Thorax	PostgreSQL	5432	TCP/TLS
Thorax	Mandible	50052	gRPC/TLS
Tarsus	Thorax	50051	gRPC/mTLS

Configuration

Cluster Configuration

Each Thorax instance requires cluster configuration:

thorax.toml

[cluster]
# Enable cluster coordination
enabled = true

# Unique instance identifier
# In Kubernetes, use metadata.name from StatefulSet
instance_id = "thorax-1"

# Redis connection for cluster state
redis_url = "redis://redis:6379"

# Heartbeat frequency (seconds)
heartbeat_interval_secs = 10

# Consider instance dead after this (seconds)
liveness_timeout_secs = 30

# Redis key namespace (allows multi-tenant Redis)
namespace = "mantis-prod"

Environment Variables

# Instance identification
THORAX__CLUSTER__ENABLED=true
THORAX__CLUSTER__INSTANCE_ID=thorax-1
THORAX__CLUSTER__REDIS_URL=redis://redis:6379
THORAX__CLUSTER__NAMESPACE=mantis-prod

# Timing
THORAX__CLUSTER__HEARTBEAT_INTERVAL_SECS=10
THORAX__CLUSTER__LIVENESS_TIMEOUT_SECS=30

Instance ID Generation

If instance_id is not configured, it’s auto-generated from the hostname:

Format: thorax-{hostname}
Example: thorax-k8s-worker-2

Note that when multiple instances share a hostname (e.g. containers on the same node), you must set THORAX__CLUSTER__INSTANCE_ID explicitly to guarantee unique identifiers.

For predictable IDs in Kubernetes, use StatefulSet pod names:

env:
  - name: THORAX__CLUSTER__INSTANCE_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

Redis Cluster State

State Structure

mantis-prod:instances:index          # SET of registered instance IDs
mantis-prod:instances:{id}           # String (JSON) with instance metadata, TTL = liveness_timeout
mantis-prod:affinity:{target_id}     # String (claim info), TTL-based
mantis-prod:affinity:by-instance:{id}  # SET of target IDs owned by an instance
mantis-prod:deployments:{uuid}       # String (JSON) with deployment state, TTL = 1h active / 24h complete
mantis-prod:deployments:active       # SET of UUIDs of running deployments
mantis-prod:deployments:by-instance:{id}  # SET of deployment UUIDs owned by instance

Instance Registration

On startup, each Thorax instance:

1. Registers itself:

   SADD mantis-prod:instances:index thorax-1
   SET_EX mantis-prod:instances:thorax-1 30 '{"instance_id":"thorax-1","started_at":1705312200,...}'

2. Sends heartbeats (every 10s):

   The heartbeat updates the last_heartbeat timestamp inside the instance JSON and rewrites the key with a refreshed TTL — there is no separate heartbeat key. Check liveness with:

   TTL mantis-prod:instances:thorax-1

3. Loads existing sessions from Mandible

Instance Cleanup

When an instance stops sending heartbeats:

1. Other instances detect missing heartbeat (TTL expired)
2. Dead instance removed from instances SET
3. Target affinities reassigned to live instances
4. In-flight deployments marked for retry

Deployment Process

Creating Instances

Docker Compose:

services:
  thorax-1:
    image: mantis/thorax:latest
    environment:
      - THORAX__CLUSTER__ENABLED=true
      - THORAX__CLUSTER__INSTANCE_ID=thorax-1
      - THORAX__CLUSTER__REDIS_URL=redis://redis:6379
    volumes:
      - ./certs:/certs:ro

  thorax-2:
    image: mantis/thorax:latest
    environment:
      - THORAX__CLUSTER__ENABLED=true
      - THORAX__CLUSTER__INSTANCE_ID=thorax-2
      - THORAX__CLUSTER__REDIS_URL=redis://redis:6379
    volumes:
      - ./certs:/certs:ro

Kubernetes StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thorax
spec:
  replicas: 3
  serviceName: thorax
  selector:
    matchLabels:
      app: thorax
  template:
    metadata:
      labels:
        app: thorax
    spec:
      containers:
        - name: thorax
          image: mantis/thorax:latest
          env:
            - name: THORAX__CLUSTER__ENABLED
              value: 'true'
            - name: THORAX__CLUSTER__INSTANCE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: THORAX__CLUSTER__REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: thorax-secrets
                  key: redis-url
          ports:
            - containerPort: 50051
              name: grpc
          volumeMounts:
            - name: certs
              mountPath: /certs
              readOnly: true

Headless Service for Target Connections

Targets need direct connections to specific Thorax instances:

apiVersion: v1
kind: Service
metadata:
  name: thorax
spec:
  clusterIP: None # Headless
  selector:
    app: thorax
  ports:
    - port: 50051
      name: grpc

Each pod is addressable as:

• thorax-0.thorax.namespace.svc.cluster.local
• thorax-1.thorax.namespace.svc.cluster.local
• thorax-2.thorax.namespace.svc.cluster.local

Scaling Operations

Scale Up

# Docker Compose - add instances
docker-compose up -d --scale thorax=3

# Kubernetes - increase replicas
kubectl scale statefulset thorax --replicas=5

New instances automatically:

1. Register in Redis
2. Start consuming from RabbitMQ
3. Accept target connections
4. Participate in deployment work

Scale Down

# Kubernetes - decrease replicas
kubectl scale statefulset thorax --replicas=2

Scale-down process:

1. Instance receives SIGTERM
2. Stops accepting new work
3. Completes in-flight deployments (graceful period)
4. Deregisters from Redis
5. Target affinities migrate to remaining instances

Rolling Updates

# Kubernetes - rolling restart
kubectl rollout restart statefulset thorax

# Watch progress
kubectl rollout status statefulset thorax

Monitoring

Instance Health

# Check active instances
redis-cli SMEMBERS mantis-prod:instances:index

# Check instance details (stored as a JSON string)
redis-cli GET mantis-prod:instances:thorax-1

# Check instance liveness (TTL on the instance key; 0 or -2 means expired/gone)
redis-cli TTL mantis-prod:instances:thorax-1

Metrics

Thorax exposes its Prometheus metrics at :9090/metrics, all under the mantis_ prefix. There are no dedicated thorax_cluster_* or thorax_heartbeat_latency_ms series; use the per-instance series below (scrape each instance as a separate target and aggregate):

Metric	Type	Description
mantis_deployments_active	Gauge	Active deployments on the instance
mantis_client_sessions_active	Gauge	Connected client sessions on the instance
mantis_client_heartbeats_total	Counter	Client heartbeats received by the instance
mantis_commands_dispatched_total	Counter	Commands dispatched by the instance

Alerts

# Prometheus AlertManager rules
groups:
  - name: thorax-cluster
    rules:
      - alert: ThoraxInstanceDown
        # 'up' is the Prometheus scrape-health series for the thorax job
        expr: count(up{job="mantis-thorax"} == 1) < 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Fewer than 2 Thorax instances are being scraped successfully'

      - alert: ThoraxNoActiveDeployments
        expr: sum(mantis_deployments_active) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'No active deployments across the Thorax cluster'

Failover Behavior

Instance Failure

Target Reconnection

When an instance fails, targets:

1. Detect connection loss
2. Re-register with Mandible to obtain a new Thorax assignment
3. Reconnect to the newly assigned instance
4. Resume operations

Default reconnection timing (exponential backoff with jitter):

• Initial retry delay: 1s
• Backoff multiplier: ×2 per attempt
• Maximum delay cap: 30s
• Maximum retries: Unlimited (retries indefinitely)

Troubleshooting

Instance Not Joining Cluster

1. Check Redis connectivity:

   redis-cli -h $REDIS_HOST ping

2. Verify configuration:

   # Check instance ID is unique
   redis-cli SMEMBERS mantis-prod:instances:index

3. Check logs:

   docker logs thorax-1 | grep -i cluster

Targets Not Reconnecting

1. Check target affinity:

   redis-cli GET mantis-prod:affinity:target_id

2. Verify instance is healthy:

   redis-cli TTL mantis-prod:instances:thorax-1

3. Force rebalance (if needed):

   redis-cli DEL mantis-prod:affinity:target_id

Split-Brain Concerns

Redis cluster state prevents split-brain:

• Instance registration is atomic
• Heartbeats use TTL for automatic cleanup
• Target assignments are consistent

Best Practices

Production Deployment

1. Minimum 2 instances for HA
2. 3+ instances for production workloads
3. Odd number for consensus operations
4. Spread across zones for fault tolerance

Configuration

1. Use StatefulSet for stable network identities
2. Configure PodDisruptionBudget for safe updates
3. Set resource limits to prevent noisy neighbors
4. Enable Redis persistence for state durability

Operations

1. Monitor heartbeat latency - Early warning of issues
2. Test failover regularly - Kill instances intentionally
3. Document recovery procedures - Runbooks for incidents
4. Automate scaling - HPA based on queue depth

Next Steps

• Load Balancing - Route traffic to instances
• Competing Consumers - RabbitMQ distribution
• Instance Affinity - Target binding strategies