Queue Topology

Queue Topology

Mantis uses a structured RabbitMQ topology for command dispatch and AI log analysis. Mandible declares the entire topology on startup. Thorax is the command consumer; the mantis.analysis.requests consumer runs inside Mandible, not a separate worker process.

Virtual host

The Ansible role creates and permissions the /mantis vhost (deploy/ansible/roles/rabbitmq/tasks/main.yml), not the CLI default /, so the vhost-scoped commands below carry -p /mantis (%2Fmantis in a management API path). Override with amqps_vhost if your deployment differs.

Management API calls on this page

These calls need the management plugin. The Compose broker has it, HTTPS-only on 15671 (management.ssl.port), so every call presents the CA and credentials. An Ansible-installed broker has no management configuration at all — enable and configure the plugin there first, or let Mandible declare the topology instead of using the manual section below. rabbitmqadmin is deliberately not used: its v1 and v2 argument grammars are incompatible and 4.x ships v2. -f is wanted here: these calls declare topology, so a 4xx should fail the script rather than print a body. Export RABBITMQ_PASSWORD first — the array captures it at definition time — then define these once per shell before running any block below.

API=https://rabbitmq.example.com:15671/api
CURL=(curl -fsS --cacert /etc/mantis/certs/rabbitmq-ca.pem
      -u "mantis:$RABBITMQ_PASSWORD" -H "content-type: application/json")

Results and events are not queue-based

RabbitMQ carries command dispatch (an optional push nudge for listen-mode agents) and AI log analysis only. Execution results flow back to Mandible over gRPC (store_command_result), not a queue — there is no mantis.results exchange. Deployment lifecycle events are delivered to clients via Server-Sent Events (the /sse endpoints), not a mantis.events exchange.

Overview

Exchanges

Exchange	Type	Purpose
mantis.commands	Direct	Route commands to Thorax by mode and priority
mantis.analysis	Direct	Route AI log-analysis jobs to Mandible’s analysis consumer
mantis.dlx	Fanout	Dead-letter exchange for failed messages

Command Exchange

The mantis.commands exchange routes commands based on target mode and priority:

mantis.commands (direct) — every command published lands on exactly one of the
three priority queues, which are what Thorax consumes:
├── command.poll.high     -> mantis.commands.priority.high
├── command.listen.high   -> mantis.commands.priority.high
├── command.poll.normal   -> mantis.commands.priority.normal
├── command.listen.normal -> mantis.commands.priority.normal
├── command.poll.low      -> mantis.commands.priority.low
└── command.listen.low    -> mantis.commands.priority.low

Poll and listen mode share a queue at each tier: the mode decides how Thorax hands the command to the agent, not which queue carries it. Every command routing key binds to exactly one queue, so a command is delivered once.

Analysis Exchange

The mantis.analysis exchange routes AI log-analysis jobs to the mantis.analysis.requests queue, consumed inside Mandible. Mantis itself queues analysis in the database; the exchange is for an external component publishing its own requests.

Results (gRPC, not a queue)

Execution results are not published to RabbitMQ. Thorax reports each command’s outcome to Mandible directly over gRPC (store_command_result), so there is no mantis.results exchange or result queue to monitor.

Events (SSE, not a queue)

Deployment lifecycle events (deployment.*, step.*, target.*) are streamed to clients over Server-Sent Events via Mandible’s /sse endpoints, not a mantis.events exchange.

Queues

Command Queues

Queue	Purpose	Consumers
mantis.commands.priority.high	High-priority commands	Thorax instances
mantis.commands.priority.normal	Normal-priority commands	Thorax instances
mantis.commands.priority.low	Low-priority commands	Thorax instances

Each Thorax instance starts one consumer per priority queue and drains them high before normal before low. These three are the only command queues Mandible declares, and the only ones to monitor.

Analysis Queue

Queue	Purpose	Consumers
mantis.analysis.requests	AI log-analysis jobs	Mandible (in-process)

Dead-Letter Queue

Queue	Purpose	Consumers
mantis.dlq	Failed/expired messages	Manual review

Routing Keys

Command Routing

Routing Key	Target Queue	Description
command.poll.high	mantis.commands.priority.high	High-priority poll commands
command.poll.normal	mantis.commands.priority.normal	Normal poll commands
command.poll.low	mantis.commands.priority.low	Low-priority poll commands
command.listen.high	mantis.commands.priority.high	High-priority listen commands
command.listen.normal	mantis.commands.priority.normal	Normal listen commands
command.listen.low	mantis.commands.priority.low	Low-priority listen commands

Queue Configuration

Queue Arguments

Default queue configuration:

# Queue durability
durable = true
auto_delete = false
exclusive = false

# Dead-lettering
x-dead-letter-exchange = mantis.dlx

# Message TTL — set unconditionally by Mandible, and it varies per queue:
#   priority.high: 1,800,000 (30 min)   priority.normal: 3,600,000 (1 hr)
#   priority.low: 7,200,000 (2 hr)      analysis: 14,400,000 (4 hr)   mantis.dlq: 604,800,000 (7 days)
x-message-ttl = 3600000  # 1 hour (priority.normal)

Priority Queues

Priority queues use max-priority argument:

# Priority queue arguments
x-max-priority = 10
x-queue-type = classic   # x-max-priority is a classic-queue argument

The command queues must stay classic

Mandible declares mantis.commands.priority.{high,normal,low} with x-max-priority, which is a classic-queue argument: RabbitMQ refuses a quorum queue declared with it. The declaration names x-queue-type: classic alongside it, so a broker-side default_queue_type = quorum does not reach these three and startup is not at the mercy of a broker setting nobody changed on Mantis’s side. A broker whose queues were declared before the type was named accepts the re-declaration: an unnamed type on an existing classic queue is equivalent to classic.

A policy that converts them is a different matter, and so is a queue that already exists as quorum: queue arguments are immutable, so RabbitMQ answers a re-declaration with PRECONDITION_FAILED and Mandible stops at topology setup with that message. To recover, remove the policy (or the broker default), delete the three queues — they hold only broker copies: every command is a row in the database’s command_queue table, which Thorax’s polling service claims through fetch_pending_commands whether or not a broker message ever arrives — and restart Mandible, which re-declares them classic:

rabbitmqctl -p /mantis clear_policy <policy-name>   # if a policy converted them
for p in high normal low; do
  rabbitmqctl -p /mantis delete_queue "mantis.commands.priority.$p"
done

Message Flow

Command Flow

Event Flow

Deployment lifecycle events are delivered to clients over Server-Sent Events, not RabbitMQ:

Dead-Letter Handling

Dead-Letter Reasons

Messages are dead-lettered when:

Reason	Description
rejected	Message explicitly rejected
expired	Message TTL exceeded
maxlen	Queue length limit reached

Retry Behaviour

max_retries is a per-message field on the command (instinct/src/queue/messages.rs), set by Mandible when it publishes — there is no [queue] setting for it. A command message gets one broker-level retry, with no delay; everything beyond it is Mandible’s to schedule:

1. On arrival the x-death count is checked first; where max_retries is above 0 and the
   count already meets it, the message is NACKed straight to the DLQ (no processing
   attempt). At max_retries = 0 this check does not apply and step 2 runs.
2. Otherwise the message is processed.
3. First processing failure (not redelivered): NACK requeue=true — it re-enters the same
   queue immediately. This is the only retry the broker itself performs.
4. A failure of a REDELIVERED message has spent that retry. `mantis.dlx` is a fanout bound
   only to `mantis.dlq`, nothing consumes that queue, and nothing dead-letters back into
   `mantis.commands.*` — so the NACK is terminal for the message and `x-death` never
   accrues on a command queue. Thorax asks Mandible to schedule another dispatch first;
   Mandible owns the command row and the authoritative retry budget, and may refuse.
5. `max_retries = 0` skips the free requeue and goes straight to the DLQ.

DLQ Management

Monitor dead-letter queue:

# List messages in DLQ
sudo rabbitmqctl -p /mantis list_queues name messages \
    | grep mantis.dlq

# Inspect DLQ messages. ack_requeue_true puts them back; ack_requeue_false discards.
"${CURL[@]}" -X POST "$API/queues/%2Fmantis/mantis.dlq/get" \
    -d '{"count": 10, "ackmode": "ack_requeue_true", "encoding": "auto"}'

# Purge DLQ (use with caution)
sudo rabbitmqctl -p /mantis purge_queue mantis.dlq

Reprocessing Failed Messages

Requeue messages from DLQ:

# Move messages back to a command queue (pick the priority the work belongs on:
# mantis.commands.priority.high, .normal or .low)
# The shovel runs inside the broker, so the plugin must be on and the paths are
# the broker's own (/etc/rabbitmq/certs), not the client material under /etc/mantis.
sudo rabbitmq-plugins enable rabbitmq_shovel rabbitmq_shovel_management

# The broker has no plain-AMQP listener, so both URIs are amqps and present a
# certificate, which `ssl_options.fail_if_no_peer_cert = true` demands.
CERTS="cacertfile=/etc/rabbitmq/certs/ca-cert.pem&certfile=/etc/rabbitmq/certs/server-cert.pem&keyfile=/etc/rabbitmq/certs/server-key.pem&verify=verify_peer"
URI="amqps://mantis:$RABBITMQ_PASSWORD@localhost:5671/%2Fmantis?$CERTS"

sudo rabbitmqctl -p /mantis set_parameter shovel reprocess \
    "{\"src-uri\": \"$URI\", \"src-queue\": \"mantis.dlq\",
      \"dest-uri\": \"$URI\", \"dest-queue\": \"mantis.commands.priority.normal\",
      \"delete-after\": \"queue-length\"}"

Topology Declaration

Automatic Setup

Mandible declares the full topology on startup. Configure the RabbitMQ connection in mandible.toml:

mandible.toml

[queue]
enabled = true
host = "localhost"
port = 5671
username = "mantis"
password = "<the mantis broker user's password>"
vhost = "/mantis"                          # defaults to "/", where the mantis user has no permissions
ca_cert_path = "/etc/mantis/certs/ca.crt"  # mandatory when enabled = true
# The shipped broker sets fail_if_no_peer_cert, so the client pair is required too:
client_cert_path = "/etc/mantis/certs/rabbitmq-client-cert.pem"
client_key_path = "/etc/mantis/certs/rabbitmq-client-key.pem"

Topology declaration is idempotent — Mandible re-declares exchanges, queues, and bindings on every startup, so no manual setup is needed. Thorax is a pure consumer and assumes the topology already exists (it retries its consumer connection until the queues appear).

Manual Setup

Declare topology manually:

# Create exchanges
for x in commands analysis; do
    "${CURL[@]}" -X PUT "$API/exchanges/%2Fmantis/mantis.$x" \
        -d '{"type": "direct", "durable": true}'
done
"${CURL[@]}" -X PUT "$API/exchanges/%2Fmantis/mantis.dlx" \
    -d '{"type": "fanout", "durable": true}'

# Create queues — five in all, with the same TTLs Mandible declares
"${CURL[@]}" -X PUT "$API/queues/%2Fmantis/mantis.commands.priority.high" \
    -d '{"durable": true, "arguments": {"x-message-ttl": 1800000, "x-dead-letter-exchange": "mantis.dlx", "x-max-priority": 10, "x-queue-type": "classic"}}'

"${CURL[@]}" -X PUT "$API/queues/%2Fmantis/mantis.commands.priority.normal" \
    -d '{"durable": true, "arguments": {"x-message-ttl": 3600000, "x-dead-letter-exchange": "mantis.dlx", "x-max-priority": 10, "x-queue-type": "classic"}}'

"${CURL[@]}" -X PUT "$API/queues/%2Fmantis/mantis.commands.priority.low" \
    -d '{"durable": true, "arguments": {"x-message-ttl": 7200000, "x-dead-letter-exchange": "mantis.dlx", "x-max-priority": 10, "x-queue-type": "classic"}}'

"${CURL[@]}" -X PUT "$API/queues/%2Fmantis/mantis.analysis.requests" \
    -d '{"durable": true, "arguments": {"x-message-ttl": 14400000, "x-dead-letter-exchange": "mantis.dlx"}}'

# The DLQ is where dead-lettering ends, so it has no dead-letter exchange of its own.
"${CURL[@]}" -X PUT "$API/queues/%2Fmantis/mantis.dlq" \
    -d '{"durable": true, "arguments": {"x-message-ttl": 604800000}}'

# Create bindings — eight in all: poll and listen at each priority tier,
# plus analysis and the fanout to the DLQ
for tier in high normal low; do
    for mode in poll listen; do
        "${CURL[@]}" -X POST "$API/bindings/%2Fmantis/e/mantis.commands/q/mantis.commands.priority.$tier" \
            -d "{\"routing_key\": \"command.$mode.$tier\"}"
    done
done

"${CURL[@]}" -X POST "$API/bindings/%2Fmantis/e/mantis.analysis/q/mantis.analysis.requests" \
    -d '{"routing_key": "analysis.request"}'

# mantis.dlx is a fanout, so its binding carries an empty routing key.
"${CURL[@]}" -X POST "$API/bindings/%2Fmantis/e/mantis.dlx/q/mantis.dlq" \
    -d '{"routing_key": ""}'

Consumer Configuration

Prefetch Count

Configure consumer prefetch:

thorax.toml

[queue]
prefetch_count = 10  # Messages to prefetch per consumer

Consumer Tags

Thorax generates unique consumer tags:

One tag is generated per instance and a priority suffix is appended per consumer, so three tags appear on the broker:

thorax-{instance_id}-{uuid}-high
thorax-{instance_id}-{uuid}-normal
thorax-{instance_id}-{uuid}-low

Acknowledgment

Mantis uses manual acknowledgment:

Action	When
ack	Message processed successfully
nack requeue=true	Temporary failure, retry immediately
nack requeue=false	Permanent failure, route to DLX

High Availability

Mirrored Queues (RabbitMQ 3.x only)

Classic queue mirroring (ha-mode) was removed in RabbitMQ 4.x and does not apply to a supported broker — Mantis requires 4.3+. The policy below is kept only for a legacy 3.x broker; on 4.x it is rejected. For HA, run a RabbitMQ cluster — see Quorum queues for what a quorum policy must not catch.

# Set HA policy for all mantis queues (RabbitMQ 3.x only)
sudo rabbitmqctl -p /mantis set_policy ha-mantis "^mantis\." \
    '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Quorum queues

A ^mantis\. policy setting {"queue-type":"quorum"} catches the three priority command queues, which Mantis declares with x-max-priority and which RabbitMQ therefore refuses as quorum queues. Scope any such policy to exclude them, or leave the queue type alone — see The command queues must stay classic.

Consumer Failover

When a Thorax instance fails:

1. Messages are redelivered to other consumers
2. In-flight messages timeout and retry
3. Health monitor switches to polling if needed

Scaling Considerations

Multiple Consumers

Scale horizontally with competing consumers:

Every Thorax instance consumes all three priority queues, so adding an instance adds a competing consumer to each tier.

Queue Sharding

The three priority tiers are the whole command topology — Mantis declares no sharded command queues, and adding one means adding both the binding and a Thorax consumer for it. Scale by adding Thorax instances to the existing tiers first.

Instance Affinity

Listen-mode targets use instance affinity:

# Command message includes the instance ID Mandible assigned at registration,
# not the configured instance name.
assigned_instance_id = "01a0b519-67f5-88c1-b926-2e2c1f3a20fd"

Troubleshooting

Queue Not Receiving Messages

1. Check binding exists:

   "${CURL[@]}" "$API/bindings/%2Fmantis" | jq '[.[] | select(.source == "mantis.commands")]'

2. Verify routing key matches:

   # Publish test message. Command routing keys are command.{poll,listen}.{high,normal,low};
   # this one lands on mantis.commands.priority.normal.
   "${CURL[@]}" -X POST "$API/exchanges/%2Fmantis/mantis.commands/publish" \
       -d '{"properties": {}, "routing_key": "command.poll.normal", "payload": "test", "payload_encoding": "string"}'

3. Check queue is not paused:

   rabbitmqctl -p /mantis list_queues name state

Messages Stuck in DLQ

1. Inspect message headers:

   "${CURL[@]}" -X POST "$API/queues/%2Fmantis/mantis.dlq/get" \
       -d '{"count": 1, "ackmode": "ack_requeue_true", "encoding": "auto"}'

2. Check x-death header for failure reason

3. Review application logs for processing errors

Consumer Not Receiving

1. Check a consumer is registered on each command queue:

   # list_consumers takes info keys, not a filter, so the queue is matched here.
   rabbitmqctl -p /mantis list_consumers queue_name consumer_tag active \
       | grep mantis.commands.priority

2. Verify prefetch is not exhausted:

   rabbitmqctl list_channels name prefetch_count messages_unacknowledged

Next Steps

• Monitoring - RabbitMQ monitoring
• TLS Configuration - Secure connections
• RabbitMQ Setup - Initial setup