Redis Availability Runbook

Redis Availability Runbook

Mandible depends on Redis for two things, with different blast radii. Because JWT revocation checks fail closed (503) when Redis is unreachable, a Redis outage is a whole-API availability incident, not merely a degraded-SSE one. Use this runbook when MandibleJwtBlocklistRedisDown or MandibleSseRedisDown fires (see JWT configuration → Redis availability dependency).

:::caution Expected startup warning: use TLS in production On startup Mandible logs Redis connection is not using TLS (redis://). Use rediss:// in production. whenever the URL is plaintext — and every shipped compose file uses redis://. The warning is expected but actionable: switch the Redis URL to rediss:// (TLS) for any production deployment. :::

Blast radius

Dependency	Metric	Posture	Impact if Redis is down
JWT revocation (blocklist)	mantis_jwt_blocklist_redis_available	Fail-closed	All JWT-authenticated REST returns 503 Service Unavailable (Lens/UI down). API-key auth is unaffected.
SSE / server-side sessions	mantis_sse_redis_up	Degrade	SSE streaming + session enforcement degraded.

/health/ready is intentionally DB-only and stays 200 during a Redis outage — do not rely on readiness to detect this; rely on the metrics/alerts above.

When MandibleJwtBlocklistRedisDown fires (critical)

1. Confirm the scope. Check mantis_jwt_blocklist_redis_available (0 = down) and rate(mantis_jwt_blocklist_redis_errors_total[5m]). If 0/elevated, JWT REST is rejecting traffic right now.
2. Check Redis itself.

   redis-cli -h <redis-host> -p 6379 ping        # expect PONG
   systemctl status redis-server                          # bare-metal
   docker compose ps redis && docker compose logs --tail=100 redis  # compose

3. Restore Redis (restart the service / replace the instance / fix the network path). The gauge (mantis_jwt_blocklist_redis_available) returns to 1 on the next successful JWT-authenticated request — recovery is immediate once traffic resumes. To confirm Redis is back independently of request traffic, check mantis_sse_redis_up, which is updated by a background probe every ~10 s regardless of request load.
4. If Redis cannot be restored quickly, make a conscious tradeoff:
   • Communicate the auth outage to users.
   • Optionally reduce the JWT access-token TTL going forward so that, after recovery, the window in which a revoked token could have been honored is small. (Mantis does not fail-open; tokens are simply rejected while Redis is down.)
5. Post-incident: Mantis ships no Redis HA — no Sentinel or cluster support, and every consumer connects to a single Redis endpoint. High availability is entirely operator-provided (e.g. a managed Redis with an HA endpoint, or Sentinel fronted by a single VIP/proxy address); Mantis simply connects to the one URL you give it. Provisioning that is how you keep a single-instance blip from rejecting all JWT traffic.

When MandibleSseRedisDown fires (warning)

SSE streaming and server-side session enforcement are degraded, but API-key and (Redis-up) JWT REST are unaffected. Restore Redis as above; no auth outage.

Why readiness is not gated on Redis

Gating /health/ready on a per-request Redis PING was tried and reverted: the probe is unauthenticated and rate-limit-exempt, so probe-flooding would exhaust the shared Redis pool (DoS amplification), and gating per-pod readiness on a shared Redis would deschedule the whole fleet on a brief blip (cascading outage). The non-gating mantis_sse_redis_up gauge gives operators the same visibility without those failure modes.