Monitoring

Monitor RabbitMQ health, performance, and message flow for Mantis deployments.

The management API is HTTPS-only on the shipped broker, so every call presents the CA and credentials. No -f: an unhealthy check answers 503 with the JSON naming the alarm, and -f would discard exactly that body. Export RABBITMQ_PASSWORD first — the array captures it at definition time — then define these once per shell before running any block below.

API=https://localhost:15671/api
CURL=(curl -sS --cacert /etc/mantis/certs/rabbitmq-ca.pem
      -u "mantis:$RABBITMQ_PASSWORD" -H "content-type: application/json")

Overview

Management Interface

Enable Management Plugin

sudo rabbitmq-plugins enable rabbitmq_management

Access Management UI

Transport	URL	Default Port
HTTPS	https://localhost:15671	15671

This applies to the Compose stack only. .rabbitmq/rabbitmq.conf sets the management.ssl.* directives and no plain-HTTP listener, and docker-compose.yml publishes only 15671 — the plugin’s plain-HTTP default of 15672 is neither configured nor exposed.

An Ansible-installed broker gets no management configuration at all: the role neither enables the plugin nor writes any management.* directive (deploy/ansible/roles/rabbitmq/templates/rabbitmq.conf.j2 configures AMQPS listeners only). Enabling the plugin there binds the plain-HTTP default 15672; configure management.ssl.* yourself before exposing it.

Key Dashboards

Tab	Metrics
Overview	Message rates, queued messages, connections
Connections	Connection count, channels, throughput
Channels	Channel state, prefetch, unacknowledged
Exchanges	Message rates per exchange
Queues	Queue depth, consumers, rates

Command-Line Monitoring

Cluster Status

# Overall cluster health
sudo rabbitmqctl cluster_status

# Node health. `node_health_check` was deprecated in 3.8 and is a no-op on the
# supported 4.x broker, so it proves nothing; `status` and the alarm list answer.
sudo rabbitmqctl status

# Cluster alarms
sudo rabbitmqctl list_alarms

Connection Monitoring

# List all connections
sudo rabbitmqctl list_connections name state channels

# Connection details
sudo rabbitmqctl list_connections \
    name peer_host peer_port state \
    channels recv_oct send_oct

# Connections by application
sudo rabbitmqctl list_connections client_properties | grep product

Channel Monitoring

# List channels
sudo rabbitmqctl list_channels name connection state

# Channel details with prefetch
sudo rabbitmqctl list_channels \
    name connection prefetch_count \
    messages_unacknowledged messages_uncommitted

Queue Monitoring

# Queue summary
sudo rabbitmqctl -p /mantis list_queues name messages consumers

# Detailed queue stats
sudo rabbitmqctl -p /mantis list_queues \
    name messages messages_ready messages_unacknowledged \
    consumers memory state

# Mantis-specific queues
sudo rabbitmqctl -p /mantis list_queues name messages consumers \
    | grep mantis

Exchange Monitoring

# List exchanges
sudo rabbitmqctl -p /mantis list_exchanges name type

# Exchange message rates. message_stats is a management-API field, not a
# rabbitmqctl info key, so the rates come from the HTTP API.
"${CURL[@]}" "$API/exchanges/%2Fmantis" \
    | jq '.[] | {name, in: .message_stats.publish_in_details.rate, out: .message_stats.publish_out_details.rate}'

Consumer Monitoring

# List consumers
sudo rabbitmqctl -p /mantis list_consumers queue_name consumer_tag channel_pid ack_required

# Mantis consumers
sudo rabbitmqctl -p /mantis list_consumers | grep thorax

Prometheus Integration

Enable Prometheus Plugin

sudo rabbitmq-plugins enable rabbitmq_prometheus

Configure Prometheus Endpoint

# Prometheus metrics endpoint
prometheus.tcp.port = 15692
prometheus.return_per_object_metrics = true

Key Metrics

Metric	Description
`rabbitmq_connections`	Total connections
`rabbitmq_channels`	Total channels
`rabbitmq_queue_messages`	Messages in queue
`rabbitmq_queue_consumers`	Consumers per queue
`rabbitmq_queue_messages_ready`	Ready for delivery
`rabbitmq_queue_messages_unacknowledged`	In-flight messages

Prometheus Scrape Config

scrape_configs:
  - job_name: 'rabbitmq'
    static_configs:
      - targets: ['rabbitmq:15692']
    metrics_path: /metrics
    scrape_interval: 15s

Per-Queue Metrics

# Enable detailed queue metrics
scrape_configs:
  - job_name: 'rabbitmq-detailed'
    static_configs:
      - targets: ['rabbitmq:15692']
    metrics_path: /metrics/detailed
    params:
      family: [queue_metrics, exchange_metrics]

Grafana Dashboards

RabbitMQ Overview Dashboard

Import dashboard ID: 10991 (RabbitMQ Overview)

Key panels:

Message rates (publish/deliver)
Queue depths
Connection counts
Memory and disk usage

Custom Mantis Dashboard

{
  "title": "Mantis RabbitMQ",
  "panels": [
    {
      "title": "Command Queue Depth",
      "targets": [
        {
          "expr": "rabbitmq_queue_messages{queue=~\"mantis.commands.*\"}"
        }
      ]
    },
    {
      "title": "Analysis Queue Depth",
      "targets": [
        {
          "expr": "rabbitmq_queue_messages{queue=\"mantis.analysis.requests\"}"
        }
      ]
    },
    {
      "title": "Dead Letter Queue",
      "targets": [
        {
          "expr": "rabbitmq_queue_messages{queue=\"mantis.dlq\"}"
        }
      ]
    },
    {
      "title": "Thorax Consumers",
      "targets": [
        {
          "expr": "sum(rabbitmq_queue_consumers{queue=~\"mantis.commands.priority.*\"})"
        }
      ]
    }
  ]
}

Alerting

Prometheus Alert Rules

groups:
  - name: rabbitmq
    rules:
      - alert: RabbitMQDown
        expr: up{job="rabbitmq"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'RabbitMQ is down'

      - alert: RabbitMQHighMemory
        expr: rabbitmq_process_resident_memory_bytes > 2147483648
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'RabbitMQ memory usage high'

      - alert: RabbitMQQueueBacklog
        expr: rabbitmq_queue_messages{queue=~"mantis.*"} > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Queue {{ $labels.queue }} has backlog'

      - alert: RabbitMQNoConsumers
        expr: sum(rabbitmq_queue_consumers{queue=~"mantis.commands.priority.*"}) == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'No consumers for command queue'

      - alert: RabbitMQDeadLetterGrowing
        expr: increase(rabbitmq_queue_messages{queue="mantis.dlq"}[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Dead letter queue growing'

      - alert: RabbitMQHighUnacked
        expr: rabbitmq_queue_messages_unacknowledged > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High unacknowledged message count'

Alertmanager Configuration

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'ops-team'
  routes:
    - match:
        severity: critical
      receiver: 'ops-team-critical'

receivers:
  - name: 'ops-team'
    email_configs:
      - to: 'ops@example.com'
  - name: 'ops-team-critical'
    pagerduty_configs:
      - service_key: '<service-key>'

Health Checks

RabbitMQ Health Endpoint

# /api/healthchecks/node was deprecated in favour of the /api/health/checks/*
# family and is gone on 4.x.

# Resource alarms in effect anywhere in the cluster
"${CURL[@]}" "$API/health/checks/alarms" | jq

# Resource alarms on this node only
"${CURL[@]}" "$API/health/checks/local-alarms" | jq

Thorax Health Integration

Thorax reports RabbitMQ health via its HTTP health endpoint on port 9090 (port 50051 is the gRPC port and is not curl-able as JSON); $METRICS_TOKEN is metrics.auth_token, when set:

curl -s -H "Authorization: Bearer $METRICS_TOKEN" http://localhost:9090/health | jq '.queue'

Response:

{
  "healthy": true,
  "mode": "RabbitMQ",
  "consecutive_failures": 0
}

Docker Health Check

services:
  rabbitmq:
    healthcheck:
      test: ['CMD', 'rabbitmq-diagnostics', '-q', 'ping']
      interval: 30s
      timeout: 10s
      retries: 3

Kubernetes Probes

livenessProbe:
  exec:
    command:
      - rabbitmq-diagnostics
      - -q
      - ping
  initialDelaySeconds: 60
  periodSeconds: 60
  timeoutSeconds: 15
readinessProbe:
  exec:
    command:
      - rabbitmq-diagnostics
      - -q
      - check_running
  initialDelaySeconds: 20
  periodSeconds: 10
  timeoutSeconds: 10

Message Tracing

Enable Firehose

For debugging message flow:

# Enable firehose (development only)
sudo rabbitmq-plugins enable rabbitmq_tracing

# Create trace
sudo rabbitmqctl -p /mantis trace_on

Message Headers

Inspect message headers for debugging:

# ack_requeue_true: the message goes back on the queue. `ack_requeue_false` ACKs it
# away instead, which on a live command queue destroys a real command.
# $API and CURL as defined at the top of this page.
"${CURL[@]}" -X POST "$API/queues/%2Fmantis/mantis.commands.priority.normal/get" \
    -d '{"count": 1, "ackmode": "ack_requeue_true", "encoding": "auto"}' | jq

# Check x-death headers for retry info
"${CURL[@]}" -X POST "$API/queues/%2Fmantis/mantis.dlq/get" \
    -d '{"count": 1, "ackmode": "ack_requeue_true", "encoding": "auto"}' \
    | jq '.[0].properties.headers["x-death"]'

Performance Metrics

Message Rates

# message_stats is a management-API field, not a rabbitmqctl info key.

# Global message rates
"${CURL[@]}" "$API/queues/%2Fmantis" \
    | jq '.[] | {name, publish: .message_stats.publish_details.rate, deliver: .message_stats.deliver_details.rate}'

# Queue-specific rates
"${CURL[@]}" "$API/queues/%2Fmantis" \
    | jq '.[] | select(.name | startswith("mantis")) | {name, messages_ready, ack: .message_stats.ack_details.rate}'

Throughput Testing

# Using PerfTest tool. The shipped broker sets `listeners.tcp = none` and
# `ssl_options.fail_if_no_peer_cert = true`, so this must be amqps on 5671 *and*
# present a client certificate. PerfTest is a JVM tool with no --ssl-* options: the
# certificate reaches it only as a PKCS#12 keystore via javax.net.ssl properties, so
# convert the PEM pair first and mount the result.
openssl pkcs12 -export -name mantis \
    -in /etc/mantis/certs/rabbitmq-client-cert.pem \
    -inkey /etc/mantis/certs/rabbitmq-client-key.pem \
    -out /etc/mantis/certs/rabbitmq-client.p12 -passout pass:changeit

keytool -importcert -noprompt -alias rabbitmq-ca -storetype PKCS12 \
    -file /etc/mantis/certs/rabbitmq-ca.pem \
    -keystore /etc/mantis/certs/rabbitmq-ca.p12 -storepass changeit

docker run -it --rm --network host \
    -v /etc/mantis/certs:/certs:ro \
    -e JAVA_OPTS="-Djavax.net.ssl.keyStore=/certs/rabbitmq-client.p12 \
        -Djavax.net.ssl.keyStoreType=PKCS12 \
        -Djavax.net.ssl.keyStorePassword=changeit \
        -Djavax.net.ssl.trustStore=/certs/rabbitmq-ca.p12 \
        -Djavax.net.ssl.trustStoreType=PKCS12 \
        -Djavax.net.ssl.trustStorePassword=changeit" \
    pivotalrabbitmq/perf-test:2.25.0 \
    --uri "amqps://mantis:$RABBITMQ_PASSWORD@localhost:5671/%2Fmantis" \
    --producers 1 --consumers 2 \
    --queue "test-queue" \
    --time 60

Memory Analysis

# Memory breakdown
sudo rabbitmqctl status | grep -A 20 memory

# Per-queue memory
sudo rabbitmqctl -p /mantis list_queues name memory

Logging

Configure Logging

# Log to console and file
log.console = true
log.console.level = info

log.file = /var/log/rabbitmq/rabbit.log
log.file.level = info
log.file.rotation.date = $D0
log.file.rotation.count = 7

# Connection events
log.connection.level = info

# Channel events
log.channel.level = warning

Log Analysis

# Recent errors
grep -i error /var/log/rabbitmq/rabbit*.log | tail -20

# Connection issues
grep -E "(connection|closed|refused)" /var/log/rabbitmq/rabbit*.log

# Authentication failures
grep -i "access refused" /var/log/rabbitmq/rabbit*.log

Structured Logging

# Enable JSON logging
log.console.formatter = json
log.file.formatter = json

Monitoring Scripts

Queue Depth Alert

#!/bin/bash
THRESHOLD=1000
QUEUE="mantis.commands.priority.normal"

DEPTH=$(sudo rabbitmqctl -p /mantis list_queues name messages \
    | grep "$QUEUE" | awk '{print $2}')

if [ "$DEPTH" -gt "$THRESHOLD" ]; then
    echo "ALERT: Queue $QUEUE depth is $DEPTH (threshold: $THRESHOLD)"
    exit 1
fi

echo "OK: Queue $QUEUE depth is $DEPTH"
exit 0

Consumer Health Check

#!/bin/bash
QUEUE="mantis.commands.priority.normal"
MIN_CONSUMERS=1

CONSUMERS=$(sudo rabbitmqctl -p /mantis list_queues name consumers \
    | grep "$QUEUE" | awk '{print $2}')

if [ "$CONSUMERS" -lt "$MIN_CONSUMERS" ]; then
    echo "ALERT: Queue $QUEUE has $CONSUMERS consumers (minimum: $MIN_CONSUMERS)"
    exit 2
fi

echo "OK: Queue $QUEUE has $CONSUMERS consumers"
exit 0

DLQ Monitor

#!/bin/bash
DLQ="mantis.dlq"
THRESHOLD=10

MESSAGES=$(sudo rabbitmqctl -p /mantis list_queues name messages \
    | grep "$DLQ" | awk '{print $2}')

if [ "$MESSAGES" -gt "$THRESHOLD" ]; then
    echo "WARNING: DLQ has $MESSAGES messages"

    # Get sample message for debugging
    curl -fsS --cacert /etc/mantis/certs/rabbitmq-ca.pem \
        -u "mantis:$RABBITMQ_PASSWORD" -H "content-type: application/json" \
        -X POST "https://localhost:15671/api/queues/%2Fmantis/$DLQ/get" \
        -d '{"count": 1, "ackmode": "ack_requeue_true", "encoding": "auto"}' 2>/dev/null

    exit 1
fi

echo "OK: DLQ has $MESSAGES messages"
exit 0

Operational Dashboards

Real-Time Monitoring

# Watch queue depths
watch -n 5 'sudo rabbitmqctl -p /mantis list_queues name messages | grep mantis'

# Watch consumers
watch -n 5 'sudo rabbitmqctl -p /mantis list_consumers | grep thorax'

# Watch connections
watch -n 5 'sudo rabbitmqctl list_connections name state | head -20'

Nagios/Icinga Integration

# NRPE check command
command[check_rabbitmq_queue]=/opt/mantis/scripts/check-queue-depth.sh
command[check_rabbitmq_consumers]=/opt/mantis/scripts/check-consumers.sh
command[check_rabbitmq_dlq]=/opt/mantis/scripts/check-dlq.sh

Troubleshooting

High Memory Usage

Check queue depths:

sudo rabbitmqctl -p /mantis list_queues name messages memory | sort -k3 -n -r | head

Check what is holding the memory. Lazy classic queues (x-queue-mode) went with 4.0, so the knob to look at is the backlog itself:
Terminal window
```
sudo rabbitmqctl -p /mantis list_queues name messages memory
```

Purge non-critical queues if needed:

sudo rabbitmqctl -p /mantis purge_queue queue_name

Message Accumulation

Check consumer health:

sudo rabbitmqctl -p /mantis list_consumers

Check for slow consumers:

sudo rabbitmqctl list_channels prefetch_count messages_unacknowledged

Review Thorax logs for processing errors

Connection Storms

Monitor connection rate:

watch -n 1 'sudo rabbitmqctl list_connections | wc -l'

Identify problematic clients:

sudo rabbitmqctl list_connections peer_host | sort | uniq -c | sort -rn

Enable connection throttling:
```
# rabbitmq.conf
connection_max = 1000
```

Next Steps

Queue Topology - Queue structure
TLS Configuration - Secure connections
RabbitMQ Setup - Initial setup