Skip to content
MantisDocs

Scaling Overview

Scaling Overview

Section titled “Scaling Overview”

Scale Mantis to handle large deployments with multiple Thorax instances, load balancing, and high availability.

Overview

Section titled “Overview”

Mantis is designed for horizontal scaling with multiple orchestration nodes:

Scaling Components

Section titled “Scaling Components”
ComponentScaling StrategyState Management
Mandible (API)Horizontal, statelessSession state in PostgreSQL
Thorax (Orchestration)Horizontal, coordinatedRedis cluster state
Lens (UI)Horizontal, statelessAPI-backed
Tarsus (Agent)Per-targetLocal state only

Architecture Patterns

Section titled “Architecture Patterns”

Single Node

Section titled “Single Node”

For development and small deployments:

  • Single Mandible + Thorax instance
  • Direct gRPC communication
  • No Redis or RabbitMQ required

Multi-Node Production

Section titled “Multi-Node Production”

For high availability and scale:

Scaling Mechanisms

Section titled “Scaling Mechanisms”

Command Distribution

Section titled “Command Distribution”

Commands flow through RabbitMQ for distributed processing:

Target Affinity

Section titled “Target Affinity”

Agents maintain connections to specific Thorax instances:

State Synchronization

Section titled “State Synchronization”

Redis coordinates state across Thorax instances:

State TypeStoragePurpose
Instance registryRedis String (JSON) + Set indexTrack live instances
Target affinityRedis String (SETNX) + SetAgent→Thorax mapping
Deployment stateRedis String (JSON)Cross-instance progress
HeartbeatsRedis TTL keysInstance liveness

Capacity Planning

Section titled “Capacity Planning”

Resource Guidelines

Section titled “Resource Guidelines”
Deployment SizeMandibleThoraxDatabaseMemory/Instance
Small (<100 targets)112 vCPU, 4GB512MB
Medium (100-500)224 vCPU, 8GB1GB
Large (500-2000)3+3+8 vCPU, 16GB2GB
Enterprise (2000+)5+5+16+ vCPU, 32GB+4GB

Scaling Triggers

Section titled “Scaling Triggers”
MetricThresholdAction
API latency p99>500msAdd Mandible instances
Deployment queue depth>100Add Thorax instances
Database connections>80% poolIncrease pool or instances
Target connections per Thorax>500Add Thorax instances

High Availability

Section titled “High Availability”

Failure Domains

Section titled “Failure Domains”

Failover Behavior

Section titled “Failover Behavior”
Component FailureImpactRecovery
Single MandibleNone (load balanced)Automatic
Single ThoraxDeployments redistribute30s (heartbeat timeout)
RabbitMQFallback to pollingAutomatic
RedisCluster state lostRebuild on reconnect
PostgreSQL PrimaryRead-only until failoverDepends on HA setup

Configuration Summary

Section titled “Configuration Summary”

Thorax Cluster Configuration

Section titled “Thorax Cluster Configuration”
thorax.toml
[cluster]
enabled = true
instance_id = "thorax-1"  # Unique per instance
redis_url = "redis://redis-cluster:6379"
heartbeat_interval_secs = 10
liveness_timeout_secs = 30
namespace = "thorax"


[queue]
enabled = true
host = "rabbitmq-cluster"
port = 5671
prefetch_count = 10
fallback_on_failure = true

Key Environment Variables

Section titled “Key Environment Variables”
VariablePurpose
THORAX__CLUSTER__ENABLEDEnable cluster mode
THORAX__CLUSTER__INSTANCE_IDUnique instance identifier (registry key)
THORAX__CLUSTER__REDIS_URLRedis connection string
THORAX__QUEUE__ENABLEDEnable RabbitMQ consumption
THORAX__QUEUE__HOSTRabbitMQ hostname

Monitoring Scaled Deployments

Section titled “Monitoring Scaled Deployments”

Key Metrics

Section titled “Key Metrics”
MetricSourceAlert Threshold
mantis_client_sessions_active (label online)Prometheus<2
mantis_command_queue_depthPrometheus>100
mantis_deployment_duration_secondsPrometheusp99 >5min

Health Checks

Section titled “Health Checks”
Terminal window
# Check Thorax cluster state
redis-cli --cluster info


# Check RabbitMQ consumers
rabbitmqctl list_consumers


# Check instance health (HTTP metrics/health server on port 9090)
curl -s http://thorax:9090/health

Best Practices

Section titled “Best Practices”

Deployment

Section titled “Deployment”
  1. Use rolling updates - Don’t restart all instances at once
  2. Monitor during scale events - Watch queue depth and latency
  3. Test failover - Regularly kill instances to verify HA
  4. Use anti-affinity - Spread instances across zones/nodes

Configuration

Section titled “Configuration”
  1. Unique instance IDs - Required for proper clustering
  2. Consistent namespaces - Use same namespace across cluster
  3. Appropriate prefetch - Balance throughput vs memory
  4. Enable fallback - RabbitMQ polling fallback for resilience

Capacity

Section titled “Capacity”
  1. Plan for burst - 3x normal capacity for deployments
  2. Monitor database connections - Primary bottleneck
  3. Right-size Redis - Cluster state is small but critical
  4. Load test before production - Validate scaling assumptions

Next Steps

Section titled “Next Steps”