Prometheus Metrics

Prometheus Metrics

Mantis exports Prometheus-compatible metrics for monitoring notification delivery, rate limiting, circuit breaker states, and audit log throughput. All metrics use the mantis_ prefix and follow Prometheus naming conventions.

Available Metrics

Notification Metrics

Metric	Type	Labels	Description
mantis_notification_deliveries_total	Counter	channel_type, status	Total notification delivery attempts
mantis_notification_delivery_duration_seconds	Histogram	channel_type	Delivery latency (buckets: 0.1s to 30s)
mantis_notification_queue_depth	Gauge	—	Number of pending notification deliveries
mantis_notification_circuit_breaker_state	Gauge	channel_id	Breaker state per channel: 0 closed, 1 open, 2 half-open
mantis_notification_events_total	Counter	event_type, event_group	Notification events emitted, by type and group
mantis_notification_dead_letter_total	Counter	channel_type, reason	Deliveries moved to dead letter queue
mantis_notification_rate_limited_total	Counter	limit_type	Deliveries rejected by rate limiter

Rate Limiting Metrics

Metric	Type	Labels	Description
mantis_rate_limit_cache_entries	Gauge	limiter	Current entries in rate limit cache
mantis_rate_limit_decisions_total	Counter	limiter, decision	Decisions by limiter (auth, general) and outcome

Audit Log Metrics

Metric	Type	Labels	Description
mantis_audit_log_events_total	Counter	result	Audit log events by result (accepted/channel_full/channel_closed)

Redis / SSE Metrics

Metric	Type	Labels	Description
mantis_jwt_blocklist_redis_available	Gauge	—	1 if the JWT-revocation Redis is reachable, 0 otherwise
mantis_jwt_blocklist_redis_errors_total	Counter	operation	Redis errors on the JWT-blocklist path
mantis_sse_redis_up	Gauge	—	1 if the SSE session Redis is up

WireGuard / Pattern Metrics

Metric	Type	Labels	Description
mantis_wireguard_overlay_allocations_total	Counter	controller	Overlay IP allocations, per controller
mantis_pattern_overload_dropped_total	Counter	—	Pattern-detection events dropped under overload

Thorax Metrics

Exposed on Thorax’s /metrics endpoint (THORAX__METRICS__BIND_ADDR, default 127.0.0.1:9090). None of the families above are Mandible’s — every metric in the tables above is exposed on Mandible’s endpoint instead, so both jobs are needed to see the whole platform.

Metric	Type	Labels	Description
mantis_tls_rate_limit_alerts_total	Counter	—	TLS handshake rate-limit alerts raised
mantis_tls_connections_refused_total	Counter	—	Connections refused by the TLS rate limiter
mantis_tls_handshake_failures_total	Counter	—	Failed TLS handshakes
mantis_tls_tracked_ips	Gauge	—	Client IPs the TLS rate limiter is tracking
mantis_deployments_started_total	Counter	—	Deployments started
mantis_deployments_completed_total	Counter	status	Deployments completed, by outcome
mantis_deployments_active	Gauge	—	Deployments currently running
mantis_deployment_duration_seconds	Histogram	—	Deployment wall-clock duration
mantis_deployment_target_count	Histogram	—	Targets per deployment
mantis_commands_dispatched_total	Counter	—	Commands dispatched to agents
mantis_commands_completed_total	Counter	status	Commands completed, by outcome
mantis_command_execution_duration_seconds	Histogram	—	Command execution duration
mantis_dispatch_attempts_total	Counter	—	Dispatch attempts
mantis_dispatch_results_total	Counter	result	Dispatch outcomes
mantis_dispatch_connection_errors_total	Counter	—	Dispatch connection errors
mantis_dispatch_duration_seconds	Histogram	—	Dispatch round-trip duration
mantis_relay_frames_dropped_total	Counter	—	WireGuard relay frames dropped

The four mantis_tls_* families are the ones Thorax Configuration and the Production Checklist tell operators to watch.

Label Values

channel_type

email, webhook, slack, teams, discord

status (delivery)

success, failed, rate_limited, circuit_open, permanent_failure, timeout, error

limiter

auth (authentication rate limiter with progressive backoff), general (API rate limiter with token bucket)

decision

allowed, rejected, locked_out (auth limiter only)

Circuit Breaker State Values

Value	State	Meaning
0	Closed	Healthy, requests flowing normally
1	Open	Failing, requests blocked
2	Half-open	Testing recovery

Where Metrics Are Exposed

Thorax serves a Prometheus endpoint at http://<thorax-host>:9090/metrics (the HTTP metrics/health server bound to 127.0.0.1:9090 by default, THORAX__METRICS__BIND_ADDR). This endpoint exposes Thorax’s own mantis_* families — gRPC request, client session/registration, command, deployment, dispatch, DB, and TLS series (see [thorax/src/metrics.rs]).

Mandible serves its Prometheus endpoint at http://<mandible-host>:9091/metrics (bound to 127.0.0.1:9091 by default; override via MANDIBLE__METRICS__BIND_ADDR, empty string to disable). This exposes Mandible’s notification, rate-limiting, and audit-log mantis_* families documented above.

Both listeners answer only to a bearer token once metrics.auth_token is set (MANDIBLE__METRICS__AUTH_TOKEN, THORAX__METRICS__AUTH_TOKEN), and a bind beyond loopback refuses to start without one. The compose stack sets both from MANTIS_METRICS_TOKEN.

Scraping Configuration

Prometheus

Scrape both endpoints in your prometheus.yml, presenting the token from a file only Prometheus can read. The alerting rules and PromQL below name Mandible families, so the mantis-mandible job is the one they depend on:

scrape_configs:
  - job_name: 'mantis-mandible'
    scrape_interval: 15s
    authorization:
      credentials_file: /etc/prometheus/mantis-metrics-token
    static_configs:
      - targets: ['mandible:9091']
    metrics_path: '/metrics'

  - job_name: 'mantis-thorax'
    scrape_interval: 15s
    authorization:
      credentials_file: /etc/prometheus/mantis-metrics-token
    static_configs:
      - targets: ['thorax:9090']
    metrics_path: '/metrics'

Grafana Agent / Alloy

metrics:
  configs:
    - name: mantis
      scrape_configs:
        - job_name: mantis-mandible
          authorization:
            credentials_file: /etc/prometheus/mantis-metrics-token
          static_configs:
            - targets: ['mandible:9091']
        - job_name: mantis-thorax
          authorization:
            credentials_file: /etc/prometheus/mantis-metrics-token
          static_configs:
            - targets: ['thorax:9090']

Useful PromQL Queries

Notification Delivery Rate

# Deliveries per second by status
rate(mantis_notification_deliveries_total[5m])

# Success rate percentage
sum(rate(mantis_notification_deliveries_total{status="success"}[5m]))
/
sum(rate(mantis_notification_deliveries_total[5m])) * 100

Delivery Latency

# P50 delivery latency
histogram_quantile(0.5, rate(mantis_notification_delivery_duration_seconds_bucket[5m]))

# P99 delivery latency
histogram_quantile(0.99, rate(mantis_notification_delivery_duration_seconds_bucket[5m]))

Queue Depth Monitoring

# Current queue depth
mantis_notification_queue_depth

# Alert if queue depth exceeds 100 for more than 5 minutes
mantis_notification_queue_depth > 100

Rate Limiting

# Rate limit rejections per second
rate(mantis_rate_limit_decisions_total{decision="rejected"}[5m])

# Lockout events (potential brute force)
increase(mantis_rate_limit_decisions_total{limiter="auth", decision="locked_out"}[1h])

Circuit Breaker Health

# Any channel with open circuit breaker
mantis_notification_circuit_breaker_state == 1

Alerting Rules

Example Prometheus Alerting Rules

groups:
  - name: mantis
    rules:
      - alert: MantisNotificationQueueBacklog
        expr: mantis_notification_queue_depth > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Notification queue backlog detected'
          description: 'Queue depth is {{ $value }} (threshold: 100)'

      - alert: MantisCircuitBreakerOpen
        expr: mantis_notification_circuit_breaker_state == 1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'Circuit breaker open for channel {{ $labels.channel_name }}'

      - alert: MantisHighDeliveryFailureRate
        expr: |
          sum(rate(mantis_notification_deliveries_total{status="failed"}[5m]))
          /
          sum(rate(mantis_notification_deliveries_total[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Notification delivery failure rate above 10%'

      - alert: MantisAuthBruteForce
        expr: increase(mantis_rate_limit_decisions_total{limiter="auth", decision="locked_out"}[15m]) > 5
        labels:
          severity: critical
        annotations:
          summary: 'Multiple auth lockouts detected -- possible brute force attack'

      - alert: MantisAuditLogDrops
        expr: increase(mantis_audit_log_events_total{result="channel_full"}[5m]) > 0
        labels:
          severity: warning
        annotations:
          summary: 'Audit log events being dropped due to full channel'

Grafana Dashboard

A recommended Grafana dashboard should include the following panels:

1. Delivery Rate — rate(mantis_notification_deliveries_total[5m]) stacked by status
2. Delivery Latency Heatmap — mantis_notification_delivery_duration_seconds_bucket
3. Queue Depth — mantis_notification_queue_depth as a time series
4. Circuit Breaker Status — mantis_notification_circuit_breaker_state as a status map
5. Rate Limit Activity — rate(mantis_rate_limit_decisions_total[5m]) by decision
6. Dead Letters — increase(mantis_notification_dead_letter_total[1h]) by reason
7. Audit Log Throughput — rate(mantis_audit_log_events_total[5m]) by result

Next Steps

• Health Endpoints — Service health monitoring
• Logging — Structured log configuration
• tokio-console — Async runtime debugging