Cluster Coordination

Cluster Coordination

Coordinate multiple Thorax instances using Redis for instance registry, target affinity, and deployment state.

Overview

Instance Registry

Purpose

The instance registry tracks all live Thorax instances in the cluster:

• Discover other instances for routing
• Detect failed instances for cleanup
• Coordinate target ownership handoff

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:instances:{id}	String (JSON)	liveness_timeout	Instance info
{ns}:instances:index	Set	None	All instance IDs

Instance Info Structure

{
  "instance_id": "thorax-node-1",
  "hostname": "thorax-1.mantis.local",
  "grpc_address": "10.0.0.1:50051",
  "started_at": 1704067200,
  "last_heartbeat": 1704067260,
  "version": "0.1.0"
}

Registration Flow

Configuration

thorax.toml

[cluster]
enabled = true
redis_url = "redis://:password@redis:6379"
namespace = "thorax"

# Heartbeat interval (default: 10s)
heartbeat_interval_secs = 10

# TTL for instance keys (default: 30s)
# Should be > 2x heartbeat_interval
liveness_timeout_secs = 30

# Instance ID (default: auto-generated)
# instance_id = "thorax-node-1"

Automatic Instance ID

If not configured, instance ID is generated from:

1. THORAX__CLUSTER__INSTANCE_ID environment variable
2. System hostname prefixed with thorax- (e.g. thorax-myhost)
3. The literal string thorax (last-resort fallback)

For predictable IDs in Kubernetes:

env:
  - name: THORAX__CLUSTER__INSTANCE_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

Target Affinity

Purpose

Target affinity ensures each listen-mode target is owned by exactly one Thorax instance:

• Prevents duplicate command dispatch
• Enables efficient direct routing
• Handles graceful ownership transfer

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:affinity:{target_id}	String	liveness_timeout	Claim info
{ns}:affinity:by-instance:{id}	Set	liveness_timeout	Targets owned by instance

Claim Protocol

Claim Value Format

{instance_id}:{claim_id}

The claim_id is a UUID that acts as a fencing token to detect stale claims.

Ownership Transfer

When a Thorax instance fails:

1. Redis TTL expires on affinity keys
2. Target reconnects to any available Thorax
3. New Thorax claims the target
4. Old affinity is cleaned up

Query Target Owner

# Find which instance owns a target
redis-cli GET "thorax:affinity:42"
# Result: thorax-1:550e8400-e29b-41d4-a716-446655440000

# List all targets owned by an instance
redis-cli SMEMBERS "thorax:affinity:by-instance:thorax-1"
# Result: 42, 43, 44

Deployment State

Purpose

Track deployment progress across Thorax instances:

• Resume deployments after failover
• Coordinate multi-target deployments
• Share step completion status

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:deployment:{id}:state	Hash	24h	Deployment progress
{ns}:deployment:{id}:targets	Set	24h	Involved targets
{ns}:deployment:{id}:step:{n}	Hash	24h	Step results

Deployment State Structure

HSET thorax:deployment:abc123:state
  status "in_progress"
  current_step 2
  total_steps 5
  started_at 1704067200
  last_update 1704067260
  owning_instance "thorax-1"

Step Results

HSET thorax:deployment:abc123:step:1
  target_42 "completed"
  target_43 "completed"
  target_44 "failed"

Deployment Handoff

When the owning Thorax fails:

1. Another instance detects stale deployment
2. Takes ownership via atomic compare-and-set
3. Resumes from last checkpoint

# Atomically claim deployment ownership
redis-cli WATCH "thorax:deployment:abc123:state"
# Check owning_instance is stale
redis-cli MULTI
redis-cli HSET "thorax:deployment:abc123:state" owning_instance "thorax-2"
redis-cli EXEC

Health Monitoring

Cluster Health Check

Thorax monitors cluster health and reports via health endpoint:

The HTTP health endpoint is served on port 9090 (port 50051 is the gRPC port):

curl -s http://localhost:9090/health | jq

{
  "status": "healthy",
  "instance_id": "thorax-node-1",
  "version": "0.1.0",
  "cluster": {
    "healthy": true,
    "active_instances": 3,
    "active_deployments": 5
  }
}

Detecting Failed Instances

# List all registered instances
redis-cli SMEMBERS "thorax:instances:index"

# Check if instance is still alive (key exists)
redis-cli EXISTS "thorax:instances:thorax-node-1"

# List instances with expired keys (stale index entries)
for id in $(redis-cli SMEMBERS "thorax:instances:index"); do
  if [ "$(redis-cli EXISTS "thorax:instances:$id")" = "0" ]; then
    echo "Stale: $id"
  fi
done

Cleanup Script

cleanup-stale-instances.sh

#!/bin/bash
NAMESPACE="thorax"

# Get all instance IDs
INSTANCES=$(redis-cli SMEMBERS "${NAMESPACE}:instances:index")

for id in $INSTANCES; do
    # Check if instance key exists
    EXISTS=$(redis-cli EXISTS "${NAMESPACE}:instances:${id}")

    if [ "$EXISTS" = "0" ]; then
        echo "Removing stale instance: $id"

        # Remove from index
        redis-cli SREM "${NAMESPACE}:instances:index" "$id"

        # Clean up affinity keys
        TARGETS=$(redis-cli SMEMBERS "${NAMESPACE}:affinity:by-instance:${id}")
        for target in $TARGETS; do
            redis-cli DEL "${NAMESPACE}:affinity:${target}"
        done
        redis-cli DEL "${NAMESPACE}:affinity:by-instance:${id}"
    fi
done

Graceful Shutdown

Deregistration

On shutdown, Thorax:

1. Stops accepting new connections
2. Releases target affinity claims
3. Deregisters from instance registry
4. Allows in-flight operations to complete

Split-Brain Prevention

Fencing Tokens

Each affinity claim includes a unique claim_id:

thorax-1:550e8400-e29b-41d4-a716-446655440000

This prevents split-brain scenarios where:

1. Thorax 1 claims target
2. Thorax 1 loses Redis connectivity
3. TTL expires
4. Thorax 2 claims target
5. Thorax 1 reconnects (stale claim)

The claim_id ensures only the current owner can release the claim.

Network Partition Handling

If a Thorax instance is partitioned from Redis:

1. Heartbeats fail
2. Instance key TTL expires
3. Affinity claims expire
4. Instance switches to degraded mode
5. Stops accepting new listen-mode connections
6. Continues processing existing work

Monitoring

Key Metrics

Metric	Query	Alert Threshold
Active instances	SCARD thorax:instances:index	< expected
Owned targets	SCARD thorax:affinity:by-instance:*	Varies
Orphaned targets	Affinity keys with dead owners	> 0

Prometheus Integration

Thorax does not currently expose cluster-specific Prometheus metrics (no mantis_cluster_* series are registered). Observe cluster state via the health endpoint (GET :9090/health, shown above) and the component logs.

Troubleshooting

Instance Not Appearing in Registry

1. Check cluster coordination is active (the cluster object is present only when clustering is enabled):

   curl -s http://localhost:9090/health | jq '.cluster'

2. Verify Redis connectivity:

   redis-cli -a password ping

3. Check for registration errors in logs:

   docker logs thorax 2>&1 | grep -i "registry\|cluster"

Target Not Being Claimed

1. Check existing claim:

   redis-cli GET "thorax:affinity:42"

2. Check claim TTL:

   redis-cli TTL "thorax:affinity:42"

3. Force release (emergency only):

   redis-cli DEL "thorax:affinity:42"

Stale Instance Entries

1. List stale entries:

   for id in $(redis-cli SMEMBERS "thorax:instances:index"); do
     if [ "$(redis-cli EXISTS "thorax:instances:$id")" = "0" ]; then
       echo "Stale: $id"
     fi
   done

2. Clean up manually:

   redis-cli SREM "thorax:instances:index" "stale-instance-id"

Next Steps

• Session Management - SSE session handling
• Redis Setup - Redis installation
• Instance Affinity - Scaling considerations