Scaling Overview

Scaling Overview

Scale Mantis to handle large deployments with multiple Thorax instances, load balancing, and high availability.

Overview

Mantis is designed for horizontal scaling with multiple orchestration nodes:

Scaling Components

Component	Scaling Strategy	State Management
Mandible (API)	Horizontal, stateless	Session state in PostgreSQL
Thorax (Orchestration)	Horizontal, coordinated	Redis cluster state
Lens (UI)	Horizontal, stateless	API-backed
Tarsus (Agent)	Per-target	Local state only

Architecture Patterns

Single Node

For development and small deployments:

• Single Mandible + Thorax instance
• Direct gRPC communication
• No Redis or RabbitMQ required

Multi-Node Production

For high availability and scale:

Scaling Mechanisms

Command Distribution

Both queueing and clustering are opt-in

RabbitMQ is disabled by default — Mantis falls back to database polling, and the flow below does not apply until you set queue.enabled = true. Multi-instance Thorax additionally requires THORAX__CLUSTER__ENABLED=true; without it, running several instances does not give you coordinated failover.

With queueing enabled, commands flow through RabbitMQ for distributed processing:

Target Affinity

Agents maintain connections to specific Thorax instances:

State Synchronization

Redis coordinates state across Thorax instances:

State Type	Storage	Purpose
Instance registry	Redis String (JSON) + Set index	Track live instances
Target affinity	Redis String (SETNX) + Set	Agent→Thorax mapping
Deployment state	Redis String (JSON)	Cross-instance progress
Heartbeats	Redis TTL keys	Instance liveness

Capacity Planning

:::note Unvalidated starting points The tables in this section are rules of thumb for sizing a first deployment, not measured limits — Mantis ships no benchmark or constant behind these numbers. Treat them as starting points and adjust from your own metrics.

Two limits that do exist in the product, and are the real ceilings to watch:

• database.max_connections (default 10) — the effective throughput ceiling for a single Mandible replica. Raise it (and PostgreSQL’s max_connections) before adding replicas if the DB pool is the bottleneck.
• internal_grpc.max_connections (default 100) — the agent-registration ceiling per Mandible. A fleet larger than this needs the value raised or more Mandible replicas. :::

Resource Guidelines

Deployment Size	Mandible	Thorax	Database	Memory/Instance
Small (<100 targets)	1	1	2 vCPU, 4GB	512MB
Medium (100-500)	2	2	4 vCPU, 8GB	1GB
Large (500-2000)	3+	3+	8 vCPU, 16GB	2GB
Enterprise (2000+)	5+	5+	16+ vCPU, 32GB+	4GB

Scaling Triggers

Suggested (not shipped) starting thresholds — Mantis ships no alert rules, so these are examples to adapt, not defaults:

Metric	Suggested threshold	Action
API latency p99	>500ms	Add Mandible instances
Deployment queue depth	>100	Add Thorax instances
Database connections	>80% pool	Increase pool or instances
Target connections per Thorax	>500	Add Thorax instances

High Availability

Failure Domains

Failover Behavior

Component Failure	Impact	Recovery
Single Mandible	None (load balanced)	Automatic
Single Thorax	Targets reassigned; in-flight deployments failed	60s liveness, checked every 30s
RabbitMQ	Fallback to polling	Automatic
Redis	Cluster state lost	Rebuild on reconnect
PostgreSQL Primary	Read-only until failover	Depends on HA setup

Configuration Summary

Thorax Cluster Configuration

thorax.toml

[cluster]
enabled = true
instance_id = "thorax-1"  # Unique per instance
redis_url = "redis://redis-cluster:6379"
heartbeat_interval_secs = 10
liveness_timeout_secs = 30
namespace = "thorax"

[queue]
enabled = true
host = "rabbitmq-cluster"
port = 5671
prefetch_count = 10
fallback_on_failure = true

Key Environment Variables

Variable	Purpose
THORAX__CLUSTER__ENABLED	Enable cluster mode
THORAX__CLUSTER__INSTANCE_ID	Unique instance identifier (registry key)
THORAX__CLUSTER__REDIS_URL	Redis connection string
THORAX__QUEUE__ENABLED	Enable RabbitMQ consumption
THORAX__QUEUE__HOST	RabbitMQ hostname

Monitoring Scaled Deployments

Key Metrics

Both Thorax (:9090/metrics) and Mandible (:9091/metrics) expose Prometheus endpoints. These thresholds are example values, not shipped alert rules — author your own based on your baseline:

Metric	Source	Example threshold
mantis_client_sessions_active{status="online"} (label key is status)	Prometheus (Thorax /metrics)	<2
mantis_command_queue_depth	Prometheus	>100
mantis_deployment_duration_seconds	Prometheus	p99 >5min

Health Checks

# Check Thorax cluster state. The registry is plain Redis keys (not Redis
# Cluster), so `--cluster info` reports nothing about Thorax. List the live
# instances from the registry index set instead:
redis-cli SMEMBERS mantis-prod:instances:index

# Check RabbitMQ consumers
rabbitmqctl list_consumers

# Check instance health (HTTP metrics/health server on port 9090)
curl -s http://thorax:9090/health

Best Practices

Deployment

1. Use rolling updates - Don’t restart all instances at once
2. Monitor during scale events - Watch queue depth and latency
3. Test failover - Regularly kill instances to verify HA
4. Use anti-affinity - Spread instances across zones/nodes

Configuration

1. Unique instance IDs - Required for proper clustering
2. Consistent namespaces - Use same namespace across cluster
3. Appropriate prefetch - Balance throughput vs memory
4. Enable fallback - RabbitMQ polling fallback for resilience

Capacity

1. Plan for burst - provision headroom above steady-state for deployment spikes (size it from your own peak, not a fixed multiple)
2. Monitor database connections - Primary bottleneck
3. Right-size Redis - Cluster state is small but critical
4. Load test before production - Validate scaling assumptions

Next Steps

• Multi-Thorax Deployment - Deploy multiple orchestrators
• Load Balancing - Configure load balancers
• Competing Consumers - RabbitMQ configuration
• Instance Affinity - Target-to-instance binding