Connection Problems

Connection Problems

This guide covers network connectivity issues between Mantis components. The typical communication flow is:

Tarsus --[register/heartbeat]--> Mandible --[dispatch]--> Thorax --[execute]--> Tarsus

Agent<->server gRPC uses mTLS. Note the REST API on port 3000 is plain HTTP by default (TLS is opt-in via [server.tls], usually terminated at a reverse proxy), so connectivity problems may be network-level or TLS-level.

Diagnosing Connectivity

Step 1: Identify Which Connection Is Failing

Connection	Protocol	Default Port	Check Command
Client to Mandible (API)	HTTPS	3000	curl http://host:3000/api/v1/health # https only if [server.tls] is configured
Mandible to Thorax	gRPC (mTLS)	50051	Check /api/v1/health/grpc
Thorax to Mandible (internal)	gRPC (mTLS)	50052	Check Thorax logs for connection errors. Listener is off by default
Tarsus to Mandible (register)	gRPC (mTLS)	50052	Check Tarsus logs for registration errors. Listener is off by default
Tarsus to assigned Thorax (poll mode)	gRPC (mTLS)	50051	Check execution logs. Poll mode only — Tarsus dials Thorax to fetch work.
Thorax to assigned Tarsus (listen mode)	gRPC (mTLS)	9342	Check execution logs. Listen mode — Thorax dials the agent on 9342 and pushes work.
Mandible to PostgreSQL	TCP	5432	psql $DATABASE_URL -c "SELECT 1"
Mandible to RabbitMQ	AMQP/TLS	5671	Check queue health in Mandible logs

Step 2: Test Network Connectivity

# Test basic TCP connectivity
nc -zv thorax-host 50051

# Test with timeout
timeout 5 bash -c "echo > /dev/tcp/thorax-host/50051" && echo "Open" || echo "Closed"

# Check DNS resolution
nslookup thorax-host
dig thorax-host

Step 3: Test TLS Connectivity

# Test TLS handshake
openssl s_client -connect thorax-host:50051 \
  -cert /etc/mantis/certs/dispatch-cert.pem \
  -key /etc/mantis/certs/dispatch-key.pem \
  -CAfile /etc/mantis/certs/ca.crt

# Check certificate details
openssl s_client -connect thorax-host:50051 < /dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Common Connection Problems

Firewall Blocking Traffic

Symptom: Connection times out rather than being refused.

# Check if port is reachable
nc -zv -w 5 target-host 3000

# Check local firewall rules (Linux)
iptables -L -n | grep -E "(3000|50051|50052)"

# Check firewall status
ufw status        # Ubuntu
firewall-cmd --list-all  # RHEL/CentOS

Fix: Open required ports between components:

From	To	Port	Purpose
Load balancer	Mandible	3000	API traffic
Mandible	Thorax	50051	Orchestration gRPC
Thorax	Mandible	50052	Internal gRPC (off by default)
Thorax	Tarsus	9342	Command execution (listen mode)
Tarsus	Mandible	50052	Registration/heartbeat (gRPC)
Mandible	PostgreSQL	5432	Database
Mandible	RabbitMQ	5671	Message queue

DNS Resolution Failures

Symptom: Name or service not known (resolver output, not a Mantis log line) or could not resolve hostname in logs.

# Verify DNS resolution
nslookup mandible-host
dig mandible-host +short

# Check /etc/hosts for local overrides
cat /etc/hosts | grep mantis

# In Docker, check service names resolve
docker compose exec thorax nslookup mandible

Fix: Ensure hostnames in configuration match DNS or /etc/hosts entries. In Docker Compose, use service names (e.g., mandible, thorax).

Connection Refused

Symptom: Connection refused error immediately.

This means the port is reachable but nothing is listening.

1. Verify the target service is running
2. Check the service is binding to the expected address (not just 127.0.0.1 when accessed remotely)
3. Confirm the port matches the configuration

# Check what's listening on the expected port
ss -tlnp | grep 3000
# or
netstat -tlnp | grep 3000

gRPC Connection Issues

Symptom: Mandible logs show gRPC operation failed or circuit breaker opens.

# Check gRPC health. The circuit_state and failure_count fields are only
# returned to authenticated callers, so pass a token; an anonymous request
# shows only the overall status with an empty circuit_state.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/health/grpc | jq

# If circuit breaker is open, check failure count
# The circuit breaker will attempt recovery in half-open state

Common gRPC causes:

• Thorax is not running or crashed
• TLS certificate mismatch between Mandible and Thorax
• Network partition between Mandible and Thorax hosts
• Thorax is overloaded and not responding to health checks

Tarsus Registration Failures

Symptom: Tarsus cannot register with Mandible.

1. Confirm Mandible’s internal gRPC server is enabled. It is off by default, so on a stock Mandible nothing listens on 50052 at all and every other check below is chasing the wrong layer. Mandible logs Internal gRPC server is disabled at startup when it is off. Turn it on with [internal_grpc] enabled = true in mandible.toml, or MANDIBLE__INTERNAL_GRPC__ENABLED=true
2. Verify Tarsus can reach Mandible’s internal gRPC port (default 50052)
3. Check Tarsus certificate is signed by the trusted CA
4. Review Tarsus logs for specific error messages

# Is the listener up at all? A closed port here usually means enabled = false,
# not a firewall.
journalctl -u mandible | grep "Internal gRPC server is disabled"
ss -tlnp | grep 50052

# Test gRPC reachability from the Tarsus host (registration uses gRPC, not REST)
nc -zv mandible-host 50052

Docker-Specific Issues

Container Network Isolation

Symptom: Services cannot reach each other in Docker Compose.

# Verify all services are on the same network
docker network inspect mantis_default

# Test connectivity between containers
docker compose exec mandible ping thorax
docker compose exec mandible nc -zv thorax 50051

Port Mapping Conflicts

Symptom: port already allocated (Docker output, not a Mantis log line) when starting containers.

# Find what's using the port
lsof -i :3000
# or
ss -tlnp | grep 3000

# Use alternative port mappings in docker-compose
# ports:
#   - "3001:3000"

Next Steps

• Certificate Issues — TLS-specific troubleshooting
• Queue Issues — RabbitMQ connectivity
• Common Issues — General troubleshooting