Queue Issues

Queue Issues

Mantis uses RabbitMQ for command dispatch from Mandible to Thorax. When RabbitMQ is unavailable, the system automatically falls back to a polling-based dispatch mode. This guide covers troubleshooting both queue connectivity and dispatch behavior.

Dispatch Modes

Mantis supports two dispatch modes, selected automatically based on RabbitMQ health:

Mode	Description	When Used
RabbitMQ	Commands published to queue, consumed by Thorax	RabbitMQ healthy
Polling	Commands stored in database, Thorax polls the API	RabbitMQ unavailable or not configured

The QueueHealthMonitor continuously checks RabbitMQ connectivity and switches modes as needed. This failover is transparent to deployment operations.

RabbitMQ Connectivity

Check Connection Status

# Check Mandible logs for queue health
journalctl -u mantis-mandible | grep -i "queue_health\|dispatch_mode\|rabbitmq" | tail -20

# Check RabbitMQ is running
systemctl status rabbitmq-server
# or
docker compose ps rabbitmq

# Test AMQP connectivity
# (RabbitMQ management plugin must be enabled)
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.message_stats'

Common Connection Errors

connection refused on port 5671/5672

RabbitMQ is not running or not listening on the expected port.

# Check which ports RabbitMQ is listening on
ss -tlnp | grep -E "(5671|5672|15672)"

# Verify RabbitMQ configuration
rabbitmqctl status

TLS handshake failed

When using AMQPS (TLS), the certificate configuration may be incorrect.

# Test TLS connection to RabbitMQ
openssl s_client -connect localhost:5671 \
  -CAfile /etc/mantis/certs/ca.crt

access refused / authentication failed

AMQP credentials are incorrect or the user lacks permissions.

# Verify user exists and has correct permissions
rabbitmqctl list_users
rabbitmqctl list_permissions -p /

Dispatch Queue Issues

Pending Dispatches Not Processing

The DispatchQueueProcessor runs in Mandible and processes pending dispatch requests. It has the following behavior:

Parameter	Value	Description
Max retries	10	Maximum retry attempts per dispatch
TTL	24 hours	Requests expire after this duration

Diagnosis:

# Check for dispatch queue activity
journalctl -u mantis-mandible | grep -i "dispatch_queue_processor" | tail -20

# Look for retry exhaustion or TTL expiry (both marked as Failed)
journalctl -u mantis-mandible | grep -i "max_retries\|expired\|failed" | tail -10

Common causes:

• No Thorax instance online to receive dispatches
• All targets assigned to an offline Thorax instance
• RabbitMQ down and polling fallback not yet activated

TTL-Expired Dispatches

Dispatch requests that exceed the 24-hour TTL are marked as Failed and will not be retried. If you see many failed dispatches due to TTL expiry:

1. Check if Thorax was offline for an extended period
2. Review the ThoraxReassignmentService logs for target reassignment activity
3. Verify new deployments can be dispatched successfully

Duplicate Dispatches

In rare cases, a dispatch may be processed more than once (at-least-once delivery). The execution layer handles idempotency, but if you see duplicate deployment runs:

1. Check for RabbitMQ connectivity flapping (rapid connect/disconnect cycles)
2. Review the dispatch queue for duplicate entries
3. Check if multiple Mandible instances are running without coordination

RabbitMQ Health Monitor

The QueueHealthMonitor tracks the following metrics:

Metric	Description
is_healthy	Whether RabbitMQ is currently reachable
consecutive_failures	Number of failed health checks in a row
total_successes	Cumulative successful checks
total_failures	Cumulative failed checks

Failover Behavior

When RabbitMQ becomes unhealthy:

1. Health monitor detects a failed health check
2. Dispatch mode switches from RabbitMQ to Polling
3. Mandible stores commands in the database
4. Thorax polls Mandible’s API for pending work

When RabbitMQ recovers:

1. Health monitor detects successful connection
2. Dispatch mode switches back to RabbitMQ
3. New commands are published to the queue
4. Pending polling-mode commands complete via their current mechanism

RabbitMQ Maintenance

Queue Management

# List queues and their message counts
rabbitmqctl list_queues name messages consumers

# Purge stuck queues (use with caution)
rabbitmqctl purge_queue mantis.commands.thorax
rabbitmqctl purge_queue mantis.commands.priority.high
rabbitmqctl purge_queue mantis.commands.priority.normal
rabbitmqctl purge_queue mantis.commands.priority.low

# Check for unacknowledged messages
rabbitmqctl list_queues name messages_unacknowledged

Connection Management

# List active connections
rabbitmqctl list_connections name state

# List channels
rabbitmqctl list_channels name consumer_count messages_unacknowledged

Debugging Checklist

1. Check RabbitMQ service status
2. Verify AMQP credentials and permissions
3. Test network connectivity to RabbitMQ port
4. Review Mandible logs for queue health status
5. Check current dispatch mode (RabbitMQ vs Polling)
6. Verify TLS certificates if using AMQPS
7. Check for message backlog in RabbitMQ queues
8. Review DispatchQueueProcessor logs for errors

Next Steps

• Deployment Failures — End-to-end deployment troubleshooting
• Connection Problems — General connectivity issues
• Common Issues — Broader troubleshooting guide