Certificate Issues

Certificate Issues

This guide covers common TLS certificate problems in Mantis deployments. Mantis uses mTLS (mutual TLS) for internal communication between Mandible, Thorax, and Tarsus, so certificate misconfiguration is one of the most common sources of connectivity failures.

Symptoms

Symptom	Likely cause
Tarsus fails to register	CA certificate mismatch or expired cert
gRPC health shows “unhealthy”	Thorax mTLS handshake failing
Client certificate required from Thorax	Agent connected without a client certificate
Certificate has been revoked from every agent at once	Mandible unreachable — the revocation check fails closed, so this is an outage, not a PKI problem
TLS alert bad certificate at the client	Client cert not signed by the configured CA
certificate verify failed (OpenSSL)	CA not trusted or chain incomplete — this is OpenSSL’s wording, from openssl s_client, not a Mantis log line

Strings above marked as OpenSSL come from the tooling you run against the service. Mantis’s own messages are the ones quoted verbatim from Thorax.

A Mandible outage looks like mass revocation

Thorax re-checks certificate revocation on every gRPC call and fails closed: while Mandible is unreachable it treats every certificate as revoked. The symptom is Certificate has been revoked from every agent simultaneously.

Simultaneity is the tell. A genuine revocation affects one agent; fleet-wide revocation errors mean Mandible is down. Check Mandible before touching any certificate.

Certificate Expiry

Check Certificate Expiry

# Check a PEM certificate file
openssl x509 -in /etc/mantis/certs/server-cert.pem -noout -enddate

# Check a remote server's certificate
openssl s_client -connect mantis.example.com:3000 < /dev/null 2>/dev/null \
  | openssl x509 -noout -enddate

# Check all certificates in a directory
for cert in /etc/mantis/certs/*-cert.pem; do
  echo "$cert: $(openssl x509 -in "$cert" -noout -enddate)"
done

Renew Certificates

If certificates have expired or are approaching expiry:

1. Generate new certificates using your CA or mantisctl
2. Replace certificate files on all affected components
3. Restart the affected services

# Restart after certificate replacement
systemctl restart mandible thorax 'tarsus@*'

CA Trust Issues

Verify CA Chain

Ensure the certificate was signed by the expected CA:

# Verify certificate against CA
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/server-cert.pem

# Check the full chain
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem \
  -untrusted /etc/mantis/certs/ca-cert.pem \
  /etc/mantis/certs/server-cert.pem

Common CA Problems

Problem: unable to get local issuer certificate

The server certificate was signed by a CA that the client does not trust. Ensure all components reference the same CA certificate file.

# Each component's TLS config must reference the CA
[tls]
ca_cert_path = "/etc/mantis/certs/ca-cert.pem"
cert_path = "/etc/mantis/certs/server-cert.pem"
key_path = "/etc/mantis/certs/server-key.pem"

Problem: self-signed certificate in certificate chain

If using self-signed certificates, the CA certificate must be explicitly trusted by every component. The system trust store is not used for mTLS verification.

mTLS Handshake Failures

Diagnose Handshake Issues

# Test mTLS connection to Mandible's internal gRPC port
openssl s_client \
  -connect localhost:50052 \
  -cert /etc/mantis/certs/server-cert.pem \
  -key /etc/mantis/certs/server-key.pem \
  -CAfile /etc/mantis/certs/ca-cert.pem

# Check for TLS version compatibility
openssl s_client -connect localhost:3000 -tls1_3

Common mTLS Problems

Problem: no suitable certificate found

The client is not presenting a certificate. Verify the component’s TLS configuration includes both cert_path and key_path.

Problem: certificate required

The server requires a client certificate but none was provided. This is expected behavior — all internal Mantis communication requires mTLS.

Problem: wrong CN (Common Name)

Mantis extracts the service identity from the certificate’s Common Name. For a Tarsus agent certificate the CN matches the client_name used at registration, but several CNs are reserved and cannot be used for an agent:

• mandible (and mandible.<region>) — the system dispatch identity.
• thorax, client, and server — reserved at enrollment and rejected.

Issuing an agent cert with, say, CN=client fails. Note also that CN-based system trust applies only in CA-signed mode — in thumbprint mode it is disabled entirely, so a reserved CN grants no special trust there. When Thorax connects to Mandible it presents the certificate at [mandible].tls_cert_path as its client identity, provisioned separately from the Thorax server certificate.

Certificate File Permissions

Ensure certificate files have correct permissions:

# Private keys should only be readable by the service user
chmod 640 /etc/mantis/certs/*-key.pem
chown mantis:mantis /etc/mantis/certs/*.key

# Certificates and CA can be world-readable
chmod 640 /etc/mantis/certs/*-cert.pem

# Verify permissions
ls -la /etc/mantis/certs/

Certificate Revocation

If a certificate has been compromised, revoke it through the Lens UI or API:

# Check if a registration (and its certificate) is revoked
curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/registrations/{id} | jq '.status'
# A revoked registration will show "revoked" as the status value

# Revoke a registration via the API
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/registrations/{id}/revoke

Revoked certificates are rejected at the application layer: Thorax checks with Mandible on each gRPC request whether the client certificate has been revoked (results are cached for up to 60 seconds). If the certificate is revoked, Thorax returns a gRPC UNAUTHENTICATED error on that request — the TLS connection itself is not closed, but every subsequent call is rejected too once the cache reflects the revocation. After revocation, the affected component must be re-enrolled with a new certificate.

Debugging Checklist

1. Check certificate expiry dates
2. Verify the CA chain (openssl verify)
3. Confirm file permissions on private keys
4. Ensure all components reference the same CA certificate
5. Check the certificate CN matches the expected identity
6. Review component logs for specific TLS error messages
7. Test connectivity with openssl s_client

Next Steps

• Connection Problems — Broader connectivity troubleshooting
• Certificate Backup — Protect your certificates
• Key Rotation Runbook — Scheduled certificate rotation