Monitoring

Monitor RabbitMQ health, performance, and message flow for Mantis deployments.

Overview

Management Interface

Enable Management Plugin

sudo rabbitmq-plugins enable rabbitmq_management

Access Management UI

Environment	URL	Default Port
HTTP	http://localhost:15672	15672
HTTPS	https://localhost:15671	15671

Key Dashboards

Tab	Metrics
Overview	Message rates, queued messages, connections
Connections	Connection count, channels, throughput
Channels	Channel state, prefetch, unacknowledged
Exchanges	Message rates per exchange
Queues	Queue depth, consumers, rates

Command-Line Monitoring

Cluster Status

# Overall cluster health
sudo rabbitmqctl cluster_status

# Node health check
sudo rabbitmqctl node_health_check

# Cluster alarms
sudo rabbitmqctl list_alarms

Connection Monitoring

# List all connections
sudo rabbitmqctl list_connections name state channels

# Connection details
sudo rabbitmqctl list_connections \
    name peer_host peer_port state \
    channels recv_oct send_oct

# Connections by application
sudo rabbitmqctl list_connections client_properties | grep product

Channel Monitoring

# List channels
sudo rabbitmqctl list_channels name connection state

# Channel details with prefetch
sudo rabbitmqctl list_channels \
    name connection prefetch_count \
    messages_unacknowledged messages_uncommitted

Queue Monitoring

# Queue summary
sudo rabbitmqctl list_queues name messages consumers

# Detailed queue stats
sudo rabbitmqctl list_queues \
    name messages messages_ready messages_unacknowledged \
    consumers memory state

# Mantis-specific queues
sudo rabbitmqctl list_queues name messages consumers \
    | grep mantis

Exchange Monitoring

# List exchanges
sudo rabbitmqctl list_exchanges name type

# Exchange message rates
sudo rabbitmqctl list_exchanges name type \
    message_stats.publish_in message_stats.publish_out

Consumer Monitoring

# List consumers
sudo rabbitmqctl list_consumers queue consumer_tag channel_pid ack_required

# Mantis consumers
sudo rabbitmqctl list_consumers | grep thorax

Prometheus Integration

Enable Prometheus Plugin

sudo rabbitmq-plugins enable rabbitmq_prometheus

Configure Prometheus Endpoint

# Prometheus metrics endpoint
prometheus.tcp.port = 15692
prometheus.return_per_object_metrics = true

Key Metrics

Metric	Description
`rabbitmq_connections`	Total connections
`rabbitmq_channels`	Total channels
`rabbitmq_queue_messages`	Messages in queue
`rabbitmq_queue_consumers`	Consumers per queue
`rabbitmq_queue_messages_ready`	Ready for delivery
`rabbitmq_queue_messages_unacknowledged`	In-flight messages

Prometheus Scrape Config

scrape_configs:
  - job_name: 'rabbitmq'
    static_configs:
      - targets: ['rabbitmq:15692']
    metrics_path: /metrics
    scrape_interval: 15s

Per-Queue Metrics

# Enable detailed queue metrics
scrape_configs:
  - job_name: 'rabbitmq-detailed'
    static_configs:
      - targets: ['rabbitmq:15692']
    metrics_path: /metrics/detailed
    params:
      family: [queue_metrics, exchange_metrics]

Grafana Dashboards

RabbitMQ Overview Dashboard

Import dashboard ID: 10991 (RabbitMQ Overview)

Key panels:

Message rates (publish/deliver)
Queue depths
Connection counts
Memory and disk usage

Custom Mantis Dashboard

{
  "title": "Mantis RabbitMQ",
  "panels": [
    {
      "title": "Command Queue Depth",
      "targets": [
        {
          "expr": "rabbitmq_queue_messages{queue=~\"mantis.commands.*\"}"
        }
      ]
    },
    {
      "title": "Analysis Queue Depth",
      "targets": [
        {
          "expr": "rabbitmq_queue_messages{queue=\"mantis.analysis\"}"
        }
      ]
    },
    {
      "title": "Dead Letter Queue",
      "targets": [
        {
          "expr": "rabbitmq_queue_messages{queue=\"mantis.dlq\"}"
        }
      ]
    },
    {
      "title": "Thorax Consumers",
      "targets": [
        {
          "expr": "rabbitmq_queue_consumers{queue=\"mantis.commands.thorax\"}"
        }
      ]
    }
  ]
}

Alerting

Prometheus Alert Rules

groups:
  - name: rabbitmq
    rules:
      - alert: RabbitMQDown
        expr: up{job="rabbitmq"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'RabbitMQ is down'

      - alert: RabbitMQHighMemory
        expr: rabbitmq_process_resident_memory_bytes > 2147483648
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'RabbitMQ memory usage high'

      - alert: RabbitMQQueueBacklog
        expr: rabbitmq_queue_messages{queue=~"mantis.*"} > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Queue {{ $labels.queue }} has backlog'

      - alert: RabbitMQNoConsumers
        expr: rabbitmq_queue_consumers{queue="mantis.commands.thorax"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'No consumers for command queue'

      - alert: RabbitMQDeadLetterGrowing
        expr: increase(rabbitmq_queue_messages{queue="mantis.dlq"}[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Dead letter queue growing'

      - alert: RabbitMQHighUnacked
        expr: rabbitmq_queue_messages_unacknowledged > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High unacknowledged message count'

Alertmanager Configuration

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'ops-team'
  routes:
    - match:
        severity: critical
      receiver: 'ops-team-critical'

receivers:
  - name: 'ops-team'
    email_configs:
      - to: 'ops@example.com'
  - name: 'ops-team-critical'
    pagerduty_configs:
      - service_key: '<service-key>'

Health Checks

RabbitMQ Health Endpoint

# Basic health check
curl -s http://localhost:15672/api/healthchecks/node | jq

# Cluster health
curl -s http://localhost:15672/api/health/checks/alarms | jq

Thorax Health Integration

Thorax reports RabbitMQ health via its HTTP health endpoint on port 9090 (port 50051 is the gRPC port and is not curl-able as JSON):

curl -s http://localhost:9090/health | jq '.queue'

Response:

{
  "healthy": true,
  "mode": "RabbitMQ",
  "consecutive_failures": 0
}

Docker Health Check

services:
  rabbitmq:
    healthcheck:
      test: ['CMD', 'rabbitmq-diagnostics', '-q', 'ping']
      interval: 30s
      timeout: 10s
      retries: 3

Kubernetes Probes

livenessProbe:
  exec:
    command:
      - rabbitmq-diagnostics
      - -q
      - ping
  initialDelaySeconds: 60
  periodSeconds: 60
  timeoutSeconds: 15
readinessProbe:
  exec:
    command:
      - rabbitmq-diagnostics
      - -q
      - check_running
  initialDelaySeconds: 20
  periodSeconds: 10
  timeoutSeconds: 10

Message Tracing

Enable Firehose

For debugging message flow:

# Enable firehose (development only)
sudo rabbitmq-plugins enable rabbitmq_tracing

# Create trace
sudo rabbitmqctl trace_on

Message Headers

Inspect message headers for debugging:

# Get messages with headers
rabbitmqadmin get queue=mantis.commands.thorax count=1 ackmode=ack_requeue_false

# Check x-death headers for retry info
rabbitmqadmin get queue=mantis.dlq count=1 | jq '.payload_bytes'

Performance Metrics

Message Rates

# Global message rates
sudo rabbitmqctl list_queues name \
    message_stats.publish_details.rate \
    message_stats.deliver_details.rate

# Queue-specific rates
sudo rabbitmqctl list_queues name messages_ready \
    message_stats.ack_details.rate \
    | grep mantis

Throughput Testing

# Using PerfTest tool
docker run -it --rm --network host \
    pivotalrabbitmq/perf-test:latest \
    --uri amqp://mantis:password@localhost \
    --producers 1 --consumers 2 \
    --queue "test-queue" \
    --time 60

Memory Analysis

# Memory breakdown
sudo rabbitmqctl status | grep -A 20 memory

# Per-queue memory
sudo rabbitmqctl list_queues name memory

Logging

Configure Logging

# Log to console and file
log.console = true
log.console.level = info

log.file = /var/log/rabbitmq/rabbit.log
log.file.level = info
log.file.rotation.date = $D0
log.file.rotation.count = 7

# Connection events
log.connection.level = info

# Channel events
log.channel.level = warning

Log Analysis

# Recent errors
grep -i error /var/log/rabbitmq/rabbit*.log | tail -20

# Connection issues
grep -E "(connection|closed|refused)" /var/log/rabbitmq/rabbit*.log

# Authentication failures
grep -i "access refused" /var/log/rabbitmq/rabbit*.log

Structured Logging

# Enable JSON logging
log.console.formatter = json
log.file.formatter = json

Monitoring Scripts

Queue Depth Alert

#!/bin/bash
THRESHOLD=1000
QUEUE="mantis.commands.thorax"

DEPTH=$(sudo rabbitmqctl list_queues name messages \
    | grep "$QUEUE" | awk '{print $2}')

if [ "$DEPTH" -gt "$THRESHOLD" ]; then
    echo "ALERT: Queue $QUEUE depth is $DEPTH (threshold: $THRESHOLD)"
    exit 1
fi

echo "OK: Queue $QUEUE depth is $DEPTH"
exit 0

Consumer Health Check

#!/bin/bash
QUEUE="mantis.commands.thorax"
MIN_CONSUMERS=1

CONSUMERS=$(sudo rabbitmqctl list_queues name consumers \
    | grep "$QUEUE" | awk '{print $2}')

if [ "$CONSUMERS" -lt "$MIN_CONSUMERS" ]; then
    echo "ALERT: Queue $QUEUE has $CONSUMERS consumers (minimum: $MIN_CONSUMERS)"
    exit 2
fi

echo "OK: Queue $QUEUE has $CONSUMERS consumers"
exit 0

DLQ Monitor

#!/bin/bash
DLQ="mantis.dlq"
THRESHOLD=10

MESSAGES=$(sudo rabbitmqctl list_queues name messages \
    | grep "$DLQ" | awk '{print $2}')

if [ "$MESSAGES" -gt "$THRESHOLD" ]; then
    echo "WARNING: DLQ has $MESSAGES messages"

    # Get sample message for debugging
    rabbitmqadmin get queue=$DLQ count=1 2>/dev/null

    exit 1
fi

echo "OK: DLQ has $MESSAGES messages"
exit 0

Operational Dashboards

Real-Time Monitoring

# Watch queue depths
watch -n 5 'sudo rabbitmqctl list_queues name messages | grep mantis'

# Watch consumers
watch -n 5 'sudo rabbitmqctl list_consumers | grep thorax'

# Watch connections
watch -n 5 'sudo rabbitmqctl list_connections name state | head -20'

Nagios/Icinga Integration

# NRPE check command
command[check_rabbitmq_queue]=/opt/mantis/scripts/check-queue-depth.sh
command[check_rabbitmq_consumers]=/opt/mantis/scripts/check-consumers.sh
command[check_rabbitmq_dlq]=/opt/mantis/scripts/check-dlq.sh

Troubleshooting

High Memory Usage

Check queue depths:

sudo rabbitmqctl list_queues name messages memory | sort -k3 -n -r | head

Check for lazy queues:

sudo rabbitmqctl list_queues name arguments | grep x-queue-mode

Purge non-critical queues if needed:
Terminal window
```
sudo rabbitmqctl purge_queue queue_name
```

Message Accumulation

Check consumer health:
Terminal window
```
sudo rabbitmqctl list_consumers
```

Check for slow consumers:

sudo rabbitmqctl list_channels prefetch_count messages_unacknowledged

Review Thorax logs for processing errors

Connection Storms

Monitor connection rate:

watch -n 1 'sudo rabbitmqctl list_connections | wc -l'

Identify problematic clients:

sudo rabbitmqctl list_connections peer_host | sort | uniq -c | sort -rn

Enable connection throttling:
```
# rabbitmq.conf
connection_max = 1000
```

Next Steps

Queue Topology - Queue structure
TLS Configuration - Secure connections
RabbitMQ Setup - Initial setup