Prometheus Metrics

Prometheus Metrics

Mantis exports Prometheus-compatible metrics for monitoring notification delivery, rate limiting, circuit breaker states, and audit log throughput. All metrics use the mantis_ prefix and follow Prometheus naming conventions.

Available Metrics

Notification Metrics

Metric	Type	Labels	Description
mantis_notification_deliveries_total	Counter	channel_type, status	Total notification delivery attempts
mantis_notification_delivery_duration_seconds	Histogram	channel_type	Delivery latency (buckets: 0.1s to 30s)
mantis_notification_queue_depth	Gauge	—	Number of pending notification deliveries
mantis_notification_circuit_breaker_state	Gauge	channel_id, channel_name	Circuit breaker state per channel
mantis_notification_events_total	Counter	event_type, event_group	Total notification events emitted
mantis_notification_dead_letter_total	Counter	channel_type, reason	Deliveries moved to dead letter queue
mantis_notification_rate_limited_total	Counter	limit_type	Deliveries rejected by rate limiter

Rate Limiting Metrics

Metric	Type	Labels	Description
mantis_rate_limit_cache_entries	Gauge	limiter	Current entries in rate limit cache
mantis_rate_limit_decisions_total	Counter	limiter, decision	Rate limit decisions (allowed/rejected/locked_out)

Audit Log Metrics

Metric	Type	Labels	Description
mantis_audit_log_events_total	Counter	result	Audit log events by result (accepted/channel_full/channel_closed)

Label Values

channel_type

email, webhook, slack, teams, discord

status (delivery)

success, failed, rate_limited, circuit_open, permanent_failure, timeout, error

limiter

auth (authentication rate limiter with progressive backoff), general (API rate limiter with token bucket)

decision

allowed, rejected, locked_out (auth limiter only)

Circuit Breaker State Values

Value	State	Meaning
0	Closed	Healthy, requests flowing normally
1	Open	Failing, requests blocked
2	Half-open	Testing recovery

Where Metrics Are Exposed

Thorax serves a Prometheus endpoint at http://<thorax-host>:9090/metrics (the HTTP metrics/health server bound to 0.0.0.0:9090). This endpoint exposes Thorax’s own mantis_* families — gRPC request, client session/registration, command, deployment, dispatch, DB, and TLS series (see [thorax/src/metrics.rs]).

Mandible does not currently expose an HTTP /metrics endpoint. The notification, rate-limiting, and audit-log families documented above are recorded in Mandible’s in-process registry but are not yet served on any port, so they cannot be scraped today. Treat the notification/rate-limit/audit dashboards and alerts below as forward-looking until a Mandible metrics endpoint is wired up.

Scraping Configuration

Prometheus

Add Thorax as a scrape target in your prometheus.yml:

scrape_configs:
  - job_name: 'mantis-thorax'
    scrape_interval: 15s
    static_configs:
      - targets: ['thorax:9090']
    metrics_path: '/metrics'

Grafana Agent / Alloy

metrics:
  configs:
    - name: mantis
      scrape_configs:
        - job_name: mantis-thorax
          static_configs:
            - targets: ['thorax:9090']

Useful PromQL Queries

Notification Delivery Rate

# Deliveries per second by status
rate(mantis_notification_deliveries_total[5m])

# Success rate percentage
sum(rate(mantis_notification_deliveries_total{status="success"}[5m]))
/
sum(rate(mantis_notification_deliveries_total[5m])) * 100

Delivery Latency

# P50 delivery latency
histogram_quantile(0.5, rate(mantis_notification_delivery_duration_seconds_bucket[5m]))

# P99 delivery latency
histogram_quantile(0.99, rate(mantis_notification_delivery_duration_seconds_bucket[5m]))

Queue Depth Monitoring

# Current queue depth
mantis_notification_queue_depth

# Alert if queue depth exceeds 100 for more than 5 minutes
mantis_notification_queue_depth > 100

Rate Limiting

# Rate limit rejections per second
rate(mantis_rate_limit_decisions_total{decision="rejected"}[5m])

# Lockout events (potential brute force)
increase(mantis_rate_limit_decisions_total{limiter="auth", decision="locked_out"}[1h])

Circuit Breaker Health

# Any channel with open circuit breaker
mantis_notification_circuit_breaker_state == 1

Alerting Rules

Example Prometheus Alerting Rules

groups:
  - name: mantis
    rules:
      - alert: MantisNotificationQueueBacklog
        expr: mantis_notification_queue_depth > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Notification queue backlog detected'
          description: 'Queue depth is {{ $value }} (threshold: 100)'

      - alert: MantisCircuitBreakerOpen
        expr: mantis_notification_circuit_breaker_state == 1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'Circuit breaker open for channel {{ $labels.channel_name }}'

      - alert: MantisHighDeliveryFailureRate
        expr: |
          sum(rate(mantis_notification_deliveries_total{status="failed"}[5m]))
          /
          sum(rate(mantis_notification_deliveries_total[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Notification delivery failure rate above 10%'

      - alert: MantisAuthBruteForce
        expr: increase(mantis_rate_limit_decisions_total{limiter="auth", decision="locked_out"}[15m]) > 5
        labels:
          severity: critical
        annotations:
          summary: 'Multiple auth lockouts detected -- possible brute force attack'

      - alert: MantisAuditLogDrops
        expr: increase(mantis_audit_log_events_total{result="channel_full"}[5m]) > 0
        labels:
          severity: warning
        annotations:
          summary: 'Audit log events being dropped due to full channel'

Grafana Dashboard

A recommended Grafana dashboard should include the following panels:

1. Delivery Rate — rate(mantis_notification_deliveries_total[5m]) stacked by status
2. Delivery Latency Heatmap — mantis_notification_delivery_duration_seconds_bucket
3. Queue Depth — mantis_notification_queue_depth as a time series
4. Circuit Breaker Status — mantis_notification_circuit_breaker_state as a status map
5. Rate Limit Activity — rate(mantis_rate_limit_decisions_total[5m]) by decision
6. Dead Letters — increase(mantis_notification_dead_letter_total[1h]) by reason
7. Audit Log Throughput — rate(mantis_audit_log_events_total[5m]) by result

Next Steps

• Health Endpoints — Service health monitoring
• Logging — Structured log configuration
• tokio-console — Async runtime debugging