Cluster Coordination

Cluster Coordination

Coordinate multiple Thorax instances using Redis for instance registry, target affinity, and deployment state.

Overview

Instance Registry

Purpose

The instance registry tracks all live Thorax instances in the cluster:

• Discover other instances for routing
• Detect failed instances for cleanup
• Coordinate target ownership handoff

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:instances:{id}	String (JSON)	liveness_timeout	Instance info
{ns}:instances:index	Set	None	All instance IDs

Instance Info Structure

{
  "instance_id": "01a0b519-67f5-88c1-b926-2e2c1f3a20fd",
  "hostname": "thorax-1.mantis.local",
  "grpc_address": "10.0.0.1:50051",
  "started_at": 1704067200,
  "last_heartbeat": 1704067260,
  "version": "0.1.0"
}

Registration Flow

Configuration

thorax.toml

[cluster]
enabled = true
redis_url = "redis://:password@redis:6379"
namespace = "thorax"

# Heartbeat interval (default: 10s)
heartbeat_interval_secs = 10

# TTL for instance keys (default: 30s)
# Should be > 2x heartbeat_interval
liveness_timeout_secs = 30

# Instance ID (default: auto-generated)
# instance_id = "thorax-node-1"

Automatic Instance ID

cluster.instance_id is the name this instance registers with. If not configured it is derived from:

1. THORAX__CLUSTER__INSTANCE_ID environment variable
2. System hostname prefixed with thorax- (e.g. thorax-myhost)
3. The literal string thorax (last-resort fallback)

It is not what the Redis keys use. Before the cluster components are built, Thorax replaces it with the instance ID Mandible assigned at registration (thorax/src/state.rs), because command_queue.assigned_instance_id records that ID and a registry keyed on anything else could not answer whether a pin’s owner is alive. So the keys below carry a UUID even when this setting is a name.

For predictable IDs in Kubernetes:

env:
  - name: THORAX__CLUSTER__INSTANCE_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

Target Affinity

Purpose

Target affinity ensures each listen-mode target is owned by exactly one Thorax instance:

• Prevents duplicate command dispatch
• Enables efficient direct routing
• Handles graceful ownership transfer

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:affinity:{target_id}	String	liveness_timeout	Claim info
{ns}:affinity:by-instance:{id}	Set	liveness_timeout	Targets owned by instance

Claim Protocol

A claim is taken when a command is dispatched to a target, not when the agent connects. There is no redirect: an instance that finds the target owned elsewhere declines the dispatch and leaves the agent’s connection alone.

Claim Value Format

{instance_id}:

The whole value is the owning instance ID followed by a single : — there is nothing after the delimiter, and there is no fencing token. The trailing : is what makes ownership decidable: a claim belongs to an instance when the stored value starts with {instance_id}:, so thorax-1 cannot own thorax-10:. For the same reason, when clustering is enabled Thorax refuses to start with a cluster.instance_id that is empty or contains a :.

Ownership Transfer

On a graceful shutdown, Thorax releases every claim it holds (cluster_release_target for each target in my_targets()), so its targets can be re-claimed at once.

When a Thorax instance exits ungracefully — a crash, a SIGKILL, a lost host — the instance itself runs no release, and:

1. Its registry entry ages out on the liveness TTL. A peer’s health monitor then sweeps it: cleanup_stale removes it from the instance index and cleanup_dead_instance deletes its affinity claims (cluster/health_monitor.rs)
2. On the next dispatch to that target, another instance claims it
3. Affinity is enforced at dispatch, not at connection — the agent is not told to reconnect elsewhere

Query Target Owner

# Find which instance owns a target
redis-cli GET "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"
# Result: "01a0b519-67f5-88c1-b926-2e2c1f3a20fd:"  -- the owning instance ID and the delimiter, nothing more

# List all targets owned by an instance
redis-cli SMEMBERS "thorax:affinity:by-instance:01a0b519-67f5-88c1-b926-2e2c1f3a20fd"
# Result: 019b9450-0b91-8c10-86c3-5c2c3f6b9f04, 019b9450-1c22-8f31-9d44-6e7f8a0b1c2d

Deployment State

Purpose

Track deployment progress across Thorax instances so an operator can see which instance owns a running deployment and inspect its state.

Redis Keys

Key Pattern	Type	TTL	Description
{ns}:deployments:{uuid}	String (JSON)	1h active, 24h once terminal	The full deployment state document
{ns}:deployments:active	Set	none	UUIDs of running deployments
{ns}:deployments:by-instance:{id}	Set	none	Deployments owned by an instance

The state is a single JSON document written with SET ... EX — not a Hash, and there are no per-step keys. While a deployment is active the TTL is 3600s; when it reaches a terminal state the TTL is refreshed to 86400s.

# Read a deployment's state
redis-cli GET thorax:deployments:019b9450-0b91-8c10-86c3-5c2c3f6b9f04

# List running deployments
redis-cli SMEMBERS thorax:deployments:active

# Which deployments an instance owns
redis-cli SMEMBERS thorax:deployments:by-instance:01a0b519-67f5-88c1-b926-2e2c1f3a20fd

Ownership

The stored owner_instance_id is preserved across updates — there is no compare-and-set handoff and no takeover path. When the owning instance dies, a peer’s health monitor calls cleanup_dead_instance, which marks each of its non-terminal deployments failed with Deployment orphaned: instance <id> died and drops the by-instance set; nothing resumes them. The deployment record in Mandible is settled failed too: when the instance registers again, or, if it stays down, once its heartbeat lapses.

Health Monitoring

Cluster Health Check

Thorax monitors cluster health and reports via health endpoint:

The HTTP health endpoint is served on port 9090 (port 50051 is the gRPC port). With metrics.auth_token set, $METRICS_TOKEN below is that token:

curl -s -H "Authorization: Bearer $METRICS_TOKEN" http://localhost:9090/health | jq

{
  "status": "healthy",
  "instance_id": "01a0b519-67f5-88c1-b926-2e2c1f3a20fd",
  "version": "0.1.0",
  "cluster": {
    "healthy": true,
    "active_instances": 3,
    "active_deployments": 5
  }
}

Detecting Failed Instances

# List all registered instances
redis-cli SMEMBERS "thorax:instances:index"

# Check if instance is still alive (key exists)
redis-cli EXISTS "thorax:instances:01a0b519-67f5-88c1-b926-2e2c1f3a20fd"

# List instances with expired keys (stale index entries)
for id in $(redis-cli SMEMBERS "thorax:instances:index"); do
  if [ "$(redis-cli EXISTS "thorax:instances:$id")" = "0" ]; then
    echo "Stale: $id"
  fi
done

Cleanup Script

cleanup-stale-instances.sh

#!/bin/bash
NAMESPACE="thorax"

# Get all instance IDs
INSTANCES=$(redis-cli SMEMBERS "${NAMESPACE}:instances:index")

for id in $INSTANCES; do
    # Check if instance key exists
    EXISTS=$(redis-cli EXISTS "${NAMESPACE}:instances:${id}")

    if [ "$EXISTS" = "0" ]; then
        echo "Removing stale instance: $id"

        # Remove from index
        redis-cli SREM "${NAMESPACE}:instances:index" "$id"

        # Clean up affinity keys
        TARGETS=$(redis-cli SMEMBERS "${NAMESPACE}:affinity:by-instance:${id}")
        for target in $TARGETS; do
            redis-cli DEL "${NAMESPACE}:affinity:${target}"
        done
        redis-cli DEL "${NAMESPACE}:affinity:by-instance:${id}"
    fi
done

Graceful Shutdown

Deregistration

On shutdown, Thorax:

1. Stops accepting new connections
2. Releases target affinity claims
3. Deregisters from instance registry
4. Allows in-flight operations to complete

Behaviour on Redis Loss

Affinity release on the claim TTL, not a fencing token

An affinity claim carries no fencing token: the stored value is just {instance_id}:, and both the release path and the ownership check compare that prefix and nothing else. A graceful shutdown does release its claims — cluster_release_target is called for every target the instance owns — so those targets are free for another instance immediately. The liveness TTL is the fallback for an ungraceful exit: each affinity key carries its own TTL and is refreshed only while its owner lives, so a dead instance’s claims lapse on their own. A peer’s health monitor deleting them once the registry entry expires is an accelerator, not the only path. Either way the target is free for another instance on its next dispatch.

Redis partition: fail open

Thorax does not enter a degraded mode when it loses Redis. Affinity is checked at dispatch time, and if the affinity check cannot reach Redis after its retries, the instance proceeds with the dispatch anyway (it logs “Affinity check failed; dispatching anyway rather than dropping the command”) rather than refusing work. This favours availability: a Redis outage does not stop deployments, at the cost of the coordination that Redis would otherwise provide. There is no state that stops accepting new listen-mode connections.

Affinity is also enforced at dispatch, not at connection: an agent is never told to reconnect to a different instance. If a target is owned elsewhere, the dispatching instance sees TargetOwnedByOther and simply declines that dispatch.

Monitoring

Key Metrics

Metric	Query	Alert Threshold
Active instances	SCARD thorax:instances:index	< expected
Owned targets	SCARD thorax:affinity:by-instance:*	Varies
Orphaned targets	Affinity keys with dead owners	> 0

Prometheus Integration

Thorax does not currently expose cluster-specific Prometheus metrics (no mantis_cluster_* series are registered). Observe cluster state via the health endpoint (GET :9090/health, shown above) and the component logs.

Troubleshooting

Instance Not Appearing in Registry

1. Check cluster coordination is active (the cluster object is present only when clustering is enabled):

   curl -s -H "Authorization: Bearer $METRICS_TOKEN" http://localhost:9090/health | jq '.cluster'

2. Verify Redis connectivity:

   redis-cli -a password ping

3. Check for registration errors in logs:

   docker logs thorax 2>&1 | grep -i "registry\|cluster"

Target Not Being Claimed

1. Check existing claim:

   redis-cli GET "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"

2. Check claim TTL:

   redis-cli TTL "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"

3. Force release (emergency only):

   redis-cli DEL "thorax:affinity:019b9450-0b91-8c10-86c3-5c2c3f6b9f04"

Stale Instance Entries

1. List stale entries:

   for id in $(redis-cli SMEMBERS "thorax:instances:index"); do
     if [ "$(redis-cli EXISTS "thorax:instances:$id")" = "0" ]; then
       echo "Stale: $id"
     fi
   done

2. Clean up manually:

   redis-cli SREM "thorax:instances:index" "stale-instance-id"

Next Steps

• Session Management - SSE session handling
• Redis Setup - Redis installation
• Instance Affinity - Scaling considerations