Common Issues

Common Issues

This page covers frequently encountered issues when operating Mantis, along with diagnosis steps and solutions.

Service Will Not Start

Mandible Fails to Start

Symptom: Mandible exits immediately after launch.

Check configuration:

# Verify config file syntax
cat /etc/mantis/mandible.toml

# Check for required environment variables
echo $MANDIBLE__DATABASE__URL
echo $MANDIBLE__JWT__SECRET

Common causes:

Log message	Cause	Fix
Failed to bind to <addr>: ...	Port 3000 already bound	Stop conflicting process or change port
Connection pool exhausted: ...	PostgreSQL unreachable or saturated	Verify database URL, connectivity and database.max_connections
JWT secret must be at least 32 characters for HS256	Secret too short or missing	Set MANDIBLE__JWT__SECRET (min 32 chars)

These are the strings Mantis emits; grep for them directly. Note TLS configuration error exists only on the RabbitMQ path — a missing server certificate does not produce it.

Cannot log in, or the UI shows stale data

If Redis is unreachable, JWT-authenticated requests return 503, not 401, and SSE stops entirely. The 503 is deliberate: reporting an auth-backend outage as a token problem makes every browser tab rotate its refresh token, turning a blip into a self-amplifying storm.

So a login failure paired with 503s is a Redis incident, not a credential problem. Check Redis before investigating the user’s account.

Database Connection Failures

Symptom: /api/v1/health returns 503, logs show connection refused.

# Test database connectivity directly
psql "$DATABASE_URL" -c "SELECT 1"

# Check if PostgreSQL is running
systemctl status postgresql
# or
docker compose ps postgres

# Verify connection pool isn't exhausted
# Look for "pool exhausted" or "timeout" in Mandible logs
journalctl -u mandible --since "5 minutes ago" | grep -i pool

Authentication Problems

Cannot Log In

Symptom: Login returns 401 or the UI shows “Invalid credentials”.

1. Verify the user exists and is not disabled
2. Check for auth rate limiting (progressive backoff after failed attempts)
3. Confirm RUST_LOG includes mandible::routes::auth=debug and review logs

# Check if rate limited (look for 429 responses)
journalctl -u mandible | grep "rate_limit" | tail -5

# Rate limits reset after the lockout period expires

Token Refresh Failures

Symptom: Users are logged out unexpectedly, or API calls return 401.

• Check that the mantis_refresh_token cookie domain matches the deployment domain
• Verify JWT secret has not changed (rotating the secret invalidates all tokens)
• Ensure system clocks are synchronized (JWT expiry is time-sensitive)

Deployment Issues

Deployments Stuck in “Pending”

Symptom: Deployments remain in Pending status and never start.

1. Check if any Thorax instance is online (the overall status is shown to any caller; authenticate to also see circuit_state, failure_count, and success_count, which are withheld from anonymous requests):

curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/health/grpc | jq

2. Verify the dispatch queue processor is running (check Mandible logs for dispatch_queue entries)
3. Check if the target has an assigned Thorax instance
4. Review RabbitMQ connectivity if using queue-based dispatch

Deployments Fail Immediately

Symptom: Deployments move to Failed within seconds.

• Check the deployment logs in the Lens UI for error details
• Verify the target (Tarsus agent) is online and reachable
• Confirm the action’s command or script payload is valid
• Check that required variables are defined for the target’s environment

Connectivity Issues

Thorax Shows “Offline”

Symptom: Instance cleanup service marks Thorax as offline.

The InstanceCleanupService checks heartbeats every 30 seconds and marks instances offline after 60 seconds of silence.

1. Verify Thorax is running: systemctl status thorax
2. Check network connectivity between Mandible and Thorax
3. Review Thorax logs for gRPC connection errors
4. Verify mTLS certificates are valid and not expired

Tarsus Cannot Register

Symptom: Tarsus agent fails to connect to Mandible.

See Connection Problems for detailed network troubleshooting, and Certificate Issues for TLS-specific problems.

Performance Problems

Slow API Responses

Symptom: API calls take several seconds to respond.

# Check database query performance
# Enable debug logging for database operations
export RUST_LOG="mandible=info,diesel=debug"
systemctl restart mandible

Common causes:

• Missing database indexes (check after schema migrations)
• Connection pool exhaustion under high load
• Large result sets without pagination

High Memory Usage

Symptom: Mandible or Thorax memory usage grows over time.

• Check notification queue depth (mantis_notification_queue_depth) and rate-limit cache size (mantis_rate_limit_cache_entries). Note: these are Mandible metrics, scrapeable at Mandible’s :9091/metrics endpoint (Thorax’s are at :9090/metrics).
• Verify the RateLimiterCleanupService is running (cleans stale cache entries)

Data Issues

Missing Audit Logs

Symptom: Expected audit log entries are not appearing.

The mantis_audit_log_events_total metric records channel_full or channel_closed results when the audit channel is under pressure or has stopped. This family is recorded in Mandible’s registry, scrapeable at :9091/metrics endpoint, so monitor it via Mandible’s structured logs (audit-channel warnings) rather than a metrics scrape.

Stale Data in UI

Symptom: Lens shows outdated information.

• Hard refresh the browser (Ctrl+Shift+R)
• Check if the API returns current data directly: curl -H "Authorization: Bearer $TOKEN" https://mantis.example.com/api/v1/targets
• Verify SSE (Server-Sent Events) connection is active for real-time updates

Getting Help

If these steps do not resolve your issue:

1. Collect relevant logs with debug-level logging enabled
2. Note the exact error messages and HTTP status codes
3. Check the Mantis version (/api/v1/server/info)
4. Review related pages for specific subsystem issues

Next Steps

• Connection Problems — Network and connectivity issues
• Deployment Failures — Deployment-specific troubleshooting
• Queue Issues — RabbitMQ and dispatch queue problems
• Certificate Issues — TLS and mTLS problems