Health Endpoints

Health Endpoints

Mantis exposes several health check endpoints on the Mandible API server for monitoring service status, integrating with load balancers, and configuring Kubernetes probes.

Endpoint Summary

Endpoint	Method	Auth Required	Purpose
/api/v1/health	GET	No	Full health check including database connectivity
/api/v1/health/live	GET	No	Kubernetes liveness probe
/api/v1/health/ready	GET	No	Kubernetes readiness probe
/api/v1/health/grpc	GET	Optional	gRPC service (Thorax) connectivity status (breaker internals require auth)

Full Health Check

GET /api/v1/health

Returns the overall service health, including database connectivity. Use this endpoint for monitoring dashboards.

curl -s https://mantis.example.com/api/v1/health | jq

Healthy response (HTTP 200):

{
  "status": "ok"
}

Unhealthy response (HTTP 503):

{
  "status": "unavailable"
}

The response body contains only the status field. The detailed error reason (e.g. connection refused, pool exhausted) is logged server-side at the WARN level and is not included in the HTTP response.

This endpoint attempts to acquire a database connection from the connection pool. If the database is unreachable, under heavy load, or the pool is exhausted, it returns 503.

Liveness Probe

GET /api/v1/health/live

Returns HTTP 200 if the Mandible process is running and not deadlocked. This endpoint does not check external dependencies (database, RabbitMQ, Thorax).

curl -s -o /dev/null -w "%{http_code}" https://mantis.example.com/api/v1/health/live
# 200

Use this as the Kubernetes liveness probe. If this endpoint stops responding, the process should be restarted.

Kubernetes Configuration

livenessProbe:
  httpGet:
    path: /api/v1/health/live
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

Readiness Probe

GET /api/v1/health/ready

Returns HTTP 200 if the service is ready to accept requests, or HTTP 503 if it is not. Three gates are checked, in order:

1. Graceful shutdown — once shutdown has begun the pod is about to stop accepting, so it reports 503 however healthy its dependencies are.
2. The audit processor — a pod whose audit task has stopped logs Readiness probe failed: the audit processor has stopped and reports 503. The gate is per-pod and deliberate: an entry submitted to a closed channel is dropped, so a pod that cannot write the audit trail must not go on serving the requests that are supposed to produce one.
3. Database connectivity — a connection is acquired from the pool.

Redis is deliberately not gated: it is a single shared instance, so an outage would mark every pod NotReady at once and turn degradation into a total outage. Redis health is surfaced through metrics instead.

A pod stuck NotReady with a healthy database is almost always one of the first two gates — check the Mandible log for the audit-processor message before investigating the database.

curl -s -o /dev/null -w "%{http_code}" https://mantis.example.com/api/v1/health/ready
# 200

Use this as the Kubernetes readiness probe. When the readiness check fails, traffic should be routed away from this instance.

Kubernetes Configuration

readinessProbe:
  httpGet:
    path: /api/v1/health/ready
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 2

gRPC Health Check

GET /api/v1/health/grpc

Returns the health status of the Orchestration gRPC connection (Mandible to Thorax), including circuit breaker state.

This endpoint never returns 401: the overall status (healthy/degraded/unhealthy) is disclosed to every caller so unauthenticated monitors and load balancers can detect an orchestration outage. The breaker internals (circuit_state, failure_count, success_count) are operational reconnaissance and are only populated for authenticated callers; anonymous requests receive an empty circuit_state and zeroed counts.

# Authenticated -- includes the breaker internals:
curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/health/grpc | jq

Example response (authenticated):

{
  "orchestration": {
    "status": "healthy",
    "circuit_state": "closed",
    "failure_count": 0,
    "success_count": 42
  }
}

Example response (anonymous) — the overall status is still disclosed, but the breaker internals are withheld:

{
  "orchestration": {
    "status": "healthy",
    "circuit_state": "",
    "failure_count": 0,
    "success_count": 0
  }
}

Circuit Breaker States

State	Status	Meaning
closed	healthy	Requests flowing normally
half_open	degraded	Testing recovery after failures
open	unhealthy	Too many failures; requests blocked

When the circuit breaker is open, Mandible stops sending requests to Thorax until recovery is detected. Monitor the failure_count to identify connectivity issues before the circuit opens.

Load Balancer Integration

HAProxy

backend mantis_api
    option httpchk GET /api/v1/health/ready
    http-check expect status 200
    server mandible1 10.0.1.10:3000 check inter 5s fall 2 rise 3
    server mandible2 10.0.1.11:3000 check inter 5s fall 2 rise 3

NGINX

upstream mantis_api {
    server 10.0.1.10:3000;
    server 10.0.1.11:3000;
}

server {
    location /api/v1/health/ready {
        proxy_pass http://mantis_api;
    }
}

AWS Application Load Balancer

Configure the target group health check:

Setting	Value
Path	/api/v1/health/ready
Port	3000
Healthy threshold	3
Unhealthy threshold	2
Interval	10 seconds
Timeout	5 seconds

Monitoring Best Practices

1. Use /health/live for restart decisions — it checks only process health, avoiding unnecessary restarts during transient database issues.
2. Use /health/ready for traffic routing — it verifies the pod is not shutting down, that its audit processor is alive, and that the database is reachable.
3. Use /health as a coarse up/down signal — it returns a single status field, not detail. For operational detail call /health/grpc with a token (public, but the breaker fields need one) or Thorax’s :9090/health with its metrics bearer token.
4. Use /health/grpc for Thorax connectivity — monitor this separately to detect orchestration layer issues.
5. Alert on sustained 503 responses from /health/ready rather than individual failures to avoid false alarms during deployments.

Next Steps

• Prometheus Metrics — Instrument detailed performance monitoring
• Logging — Configure log levels and output
• Common Issues — Troubleshoot health check failures