Cluster Coordination

Cluster Coordination

Coordinate multiple Thorax instances using Redis for instance registry, target affinity, and deployment state.

Overview

Instance Registry

Purpose

The instance registry tracks all live Thorax instances in the cluster:

• Discover other instances for routing
• Detect failed instances for cleanup
• Coordinate target ownership handoff

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:instances:{id}	String (JSON)	liveness_timeout	Instance info
{ns}:instances:index	Set	None	All instance IDs

Instance Info Structure

{
  "instance_id": "thorax-node-1",
  "hostname": "thorax-1.mantis.local",
  "grpc_address": "10.0.0.1:50051",
  "started_at": 1704067200,
  "last_heartbeat": 1704067260,
  "version": "0.1.0"
}

Registration Flow

Configuration

thorax.toml

[cluster]
enabled = true
redis_url = "redis://:password@redis:6379"
namespace = "thorax"

# Heartbeat interval (default: 10s)
heartbeat_interval_secs = 10

# TTL for instance keys (default: 30s)
# Should be > 2x heartbeat_interval
liveness_timeout_secs = 30

# Instance ID (default: auto-generated)
# instance_id = "thorax-node-1"

Automatic Instance ID

If not configured, instance ID is generated from:

1. THORAX__CLUSTER__INSTANCE_ID environment variable
2. System hostname prefixed with thorax- (e.g. thorax-myhost)
3. The literal string thorax (last-resort fallback)

For predictable IDs in Kubernetes:

env:
  - name: THORAX__CLUSTER__INSTANCE_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

Target Affinity

Purpose

Target affinity ensures each listen-mode target is owned by exactly one Thorax instance:

• Prevents duplicate command dispatch
• Enables efficient direct routing
• Handles graceful ownership transfer

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:affinity:{target_id}	String	liveness_timeout	Claim info
{ns}:affinity:by-instance:{id}	Set	liveness_timeout	Targets owned by instance

Claim Protocol

Claim Value Format

{instance_id}:{claim_id}

The claim value carries a claim_id, but ownership is decided by the instance_id prefix alone — claim_id is not a fencing token and has no consumers.

Ownership Transfer

When a Thorax instance fails:

1. Its affinity claims expire on the liveness TTL (they are not actively released)
2. On its next dispatch to that target, another instance claims it
3. Affinity is enforced at dispatch, not at connection — the agent is not told to reconnect elsewhere

Query Target Owner

# Find which instance owns a target
redis-cli GET "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"
# Result: thorax-1:550e8400-e29b-41d4-a716-446655440000

# List all targets owned by an instance
redis-cli SMEMBERS "thorax:affinity:by-instance:thorax-1"
# Result: 42, 43, 44

Deployment State

Purpose

Track deployment progress across Thorax instances so an operator can see which instance owns a running deployment and inspect its state.

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:deployments:{uuid}	String (JSON)	1h active, 24h once terminal	The full deployment state document
{ns}:deployments:active	Set	none	UUIDs of running deployments
{ns}:deployments:by-instance:{id}	Set	none	Deployments owned by an instance

The state is a single JSON document written with SET ... EX — not a Hash, and there are no per-step keys. While a deployment is active the TTL is 3600s; when it reaches a terminal state the TTL is refreshed to 86400s.

# Read a deployment's state
redis-cli GET thorax:deployments:019b9450-0b91-8c10-86c3-5c2c3f6b9f04

# List running deployments
redis-cli SMEMBERS thorax:deployments:active

# Which deployments an instance owns
redis-cli SMEMBERS thorax:deployments:by-instance:thorax-1

Ownership

The stored owner_instance_id is preserved across updates — there is no compare-and-set handoff and no takeover path. If the owning instance dies, its deployment records simply expire with their TTL; another instance does not resume them. Recovery of the deployment itself is driven by Mandible’s reassignment and dispatch-retry logic, not by a Redis ownership swap.

Health Monitoring## Health Monitoring

Cluster Health Check

Thorax monitors cluster health and reports via health endpoint:

The HTTP health endpoint is served on port 9090 (port 50051 is the gRPC port):

curl -s http://localhost:9090/health | jq

{
  "status": "healthy",
  "instance_id": "thorax-node-1",
  "version": "0.1.0",
  "cluster": {
    "healthy": true,
    "active_instances": 3,
    "active_deployments": 5
  }
}

Detecting Failed Instances

# List all registered instances
redis-cli SMEMBERS "thorax:instances:index"

# Check if instance is still alive (key exists)
redis-cli EXISTS "thorax:instances:thorax-node-1"

# List instances with expired keys (stale index entries)
for id in $(redis-cli SMEMBERS "thorax:instances:index"); do
  if [ "$(redis-cli EXISTS "thorax:instances:$id")" = "0" ]; then
    echo "Stale: $id"
  fi
done

Cleanup Script

cleanup-stale-instances.sh

#!/bin/bash
NAMESPACE="thorax"

# Get all instance IDs
INSTANCES=$(redis-cli SMEMBERS "${NAMESPACE}:instances:index")

for id in $INSTANCES; do
    # Check if instance key exists
    EXISTS=$(redis-cli EXISTS "${NAMESPACE}:instances:${id}")

    if [ "$EXISTS" = "0" ]; then
        echo "Removing stale instance: $id"

        # Remove from index
        redis-cli SREM "${NAMESPACE}:instances:index" "$id"

        # Clean up affinity keys
        TARGETS=$(redis-cli SMEMBERS "${NAMESPACE}:affinity:by-instance:${id}")
        for target in $TARGETS; do
            redis-cli DEL "${NAMESPACE}:affinity:${target}"
        done
        redis-cli DEL "${NAMESPACE}:affinity:by-instance:${id}"
    fi
done

Graceful Shutdown

Deregistration

On shutdown, Thorax:

1. Stops accepting new connections
2. Releases target affinity claims
3. Deregisters from instance registry
4. Allows in-flight operations to complete

Behaviour on Redis Loss

Affinity release on the claim TTL, not a fencing token

An affinity claim carries a claim_id, but it is not a fencing token: the release path and the ownership check compare only the instance_id prefix, and claim_id has no consumers. A claim is not actively released on shutdown either — cluster_release_target exists but is never called — so a claim lapses only when the owning instance’s liveness TTL expires. Another instance then claims the target on its next dispatch.

Redis partition: fail open

Thorax does not enter a degraded mode when it loses Redis. Affinity is checked at dispatch time, and if the affinity check cannot reach Redis after its retries, the instance proceeds with the dispatch anyway (“proceeding with dispatch”) rather than refusing work. This favours availability: a Redis outage does not stop deployments, at the cost of the coordination that Redis would otherwise provide. There is no state that stops accepting new listen-mode connections.

Affinity is also enforced at dispatch, not at connection: an agent is never told to reconnect to a different instance. If a target is owned elsewhere, the dispatching instance sees TargetOwnedByOther and simply declines that dispatch.

Monitoring## Monitoring

Key Metrics

Metric	Query	Alert Threshold
Active instances	SCARD thorax:instances:index	< expected
Owned targets	SCARD thorax:affinity:by-instance:*	Varies
Orphaned targets	Affinity keys with dead owners	> 0

Prometheus Integration

Thorax does not currently expose cluster-specific Prometheus metrics (no mantis_cluster_* series are registered). Observe cluster state via the health endpoint (GET :9090/health, shown above) and the component logs.

Troubleshooting

Instance Not Appearing in Registry

1. Check cluster coordination is active (the cluster object is present only when clustering is enabled):

   curl -s http://localhost:9090/health | jq '.cluster'

2. Verify Redis connectivity:

   redis-cli -a password ping

3. Check for registration errors in logs:

   docker logs thorax 2>&1 | grep -i "registry\|cluster"

Target Not Being Claimed

1. Check existing claim:

   redis-cli GET "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"

2. Check claim TTL:

   redis-cli TTL "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"

3. Force release (emergency only):

   redis-cli DEL "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"

Stale Instance Entries

1. List stale entries:

   for id in $(redis-cli SMEMBERS "thorax:instances:index"); do
     if [ "$(redis-cli EXISTS "thorax:instances:$id")" = "0" ]; then
       echo "Stale: $id"
     fi
   done

2. Clean up manually:

   redis-cli SREM "thorax:instances:index" "stale-instance-id"

Next Steps

• Session Management - SSE session handling
• Redis Setup - Redis installation
• Instance Affinity - Scaling considerations