Certificate Issues

Certificate Issues

This guide covers common TLS certificate problems in Mantis deployments. Mantis uses mTLS (mutual TLS) for internal communication between Mandible, Thorax, and Tarsus, so certificate misconfiguration is one of the most common sources of connectivity failures.

Symptoms

Symptom	Likely cause
Tarsus fails to register	CA certificate mismatch or expired cert
gRPC health shows “unhealthy”	Thorax mTLS handshake failing
Client certificate required from Thorax	Agent connected without a client certificate
Certificate revocation status could not be verified	Mandible unreachable — the revocation check fails closed, so this is an outage, not a PKI problem
TLS alert bad certificate at the client	Client cert not signed by the configured CA
certificate verify failed (OpenSSL)	CA not trusted or chain incomplete — this is OpenSSL’s wording, from openssl s_client, not a Mantis log line

Strings above marked as OpenSSL come from the tooling you run against the service. Mantis’s own messages are the ones quoted verbatim from Thorax.

A Mandible outage looks like mass revocation

Thorax re-checks certificate revocation on every gRPC call and fails closed: while Mandible is unreachable it refuses every certificate. The two cases carry different messages on purpose — a genuine revocation says Certificate has been revoked, an unreachable Mandible says Certificate revocation status could not be verified.

Simultaneity is the tell. A genuine revocation affects one agent; fleet-wide revocation errors mean Mandible is down. Check Mandible before touching any certificate.

Certificate Expiry

Check Certificate Expiry

# Check a PEM certificate file
openssl x509 -in /etc/mantis/certs/server-cert.pem -noout -enddate

# Check a remote server's certificate. Port 3000 serves TLS only with [server.tls]
# set; otherwise name the port your proxy terminates TLS on (443)
openssl s_client -connect mantis.example.com:3000 < /dev/null 2>/dev/null \
  | openssl x509 -noout -enddate

# Check all certificates in a directory
for cert in /etc/mantis/certs/*-cert.pem; do
  echo "$cert: $(openssl x509 -in "$cert" -noout -enddate)"
done

Renew Certificates

If certificates have expired or are approaching expiry:

1. Generate new certificates using your CA or mantisctl
2. Replace certificate files on all affected components
3. Restart the affected services

# Restart after certificate replacement
systemctl restart mandible thorax 'tarsus@*'

CA Trust Issues

Verify CA Chain

Ensure the certificate was signed by the expected CA:

# Verify certificate against CA
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/server-cert.pem

# Check the full chain
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem \
  -untrusted /etc/mantis/certs/ca-cert.pem \
  /etc/mantis/certs/server-cert.pem

Common CA Problems

Problem: unable to get local issuer certificate

The server certificate was signed by a CA that the client does not trust. Ensure all components reference the same CA certificate file.

# Each component's TLS config must reference the CA
[tls]
ca_cert_path = "/etc/mantis/certs/ca-cert.pem"
cert_path = "/etc/mantis/certs/server-cert.pem"
key_path = "/etc/mantis/certs/server-key.pem"

Problem: self-signed certificate in certificate chain

If using self-signed certificates, the CA certificate must be explicitly trusted by every component. The system trust store is not used for mTLS verification.

mTLS Handshake Failures

Diagnose Handshake Issues

# Test mTLS connection to Mandible's internal gRPC port
openssl s_client \
  -connect localhost:50052 \
  -cert /etc/mantis/certs/server-cert.pem \
  -key /etc/mantis/certs/server-key.pem \
  -CAfile /etc/mantis/certs/ca-cert.pem

# Check for TLS version compatibility
openssl s_client -connect localhost:3000 -tls1_3

Common mTLS Problems

Problem: no suitable certificate found

The client is not presenting a certificate. Verify the component’s TLS configuration includes both cert_path and key_path.

Problem: certificate required

The server requires a client certificate but none was provided. This is expected behavior — all internal Mantis communication requires mTLS.

Problem: wrong CN (Common Name)

Mantis extracts the service identity from the certificate’s Common Name. For a Tarsus agent certificate the CN matches the client_name used at registration, but several CNs are reserved and cannot be used for an agent:

• mandible (and mandible.<region>) — the system dispatch identity.
• thorax, client, and server — reserved at enrollment and rejected.

The reservation matches the CN’s leading segment, taken up to the first . or - and compared case-insensitively — so thorax-abc, Mandible.eu-west and client-web-01 are all reserved, not just the bare four names. Issuing an agent cert with, say, CN=client-web-01 fails. Note also that CN-based system trust is gated on the CA signature, not on tls.auth_mode: Thorax and Mandible authenticate to each other by CA in every configuration, so a certificate that chains to the platform CA and carries the dispatch CN is trusted as the control plane in thumbprint mode too. auth_mode governs only how a Tarsus agent enrols. Only the dispatch CN is a control-plane principal — client and server are reserved at enrollment but carry no cross-tenant trust. When Thorax connects to Mandible it presents the certificate at [mandible].tls_cert_path as its client identity, provisioned separately from the Thorax server certificate.

Certificate File Permissions

Ensure certificate files have correct permissions:

# Private keys: readable by the service user only
chmod 600 /etc/mantis/certs/*-key.pem
chown mantis:mantis /etc/mantis/certs/*-key.pem

# Certificates and the CA cert are public material
chmod 644 /etc/mantis/certs/*-cert.pem

# Verify permissions
ls -la /etc/mantis/certs/

Certificate Revocation

If a certificate has been compromised, revoke it through the Lens UI or API:

# Check if a registration (and its certificate) is revoked
curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/registrations/{id} | jq '.status'
# A revoked registration will show "revoked" as the status value

# Revoke a registration via the API
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/registrations/{id}/revoke

Revoked certificates are rejected at the application layer: Thorax checks with Mandible on each gRPC request whether the client certificate has been revoked (results are cached for up to 60 seconds). If the certificate is revoked, Thorax returns a gRPC UNAUTHENTICATED error on that request — the TLS connection itself is not closed, but every subsequent call is rejected too once the cache reflects the revocation. After revocation, the affected component must be re-enrolled with a new certificate.

Debugging Checklist

1. Check certificate expiry dates
2. Verify the CA chain (openssl verify)
3. Confirm file permissions on private keys
4. Ensure all components reference the same CA certificate
5. Check the certificate CN matches the expected identity
6. Review component logs for specific TLS error messages
7. Test connectivity with openssl s_client

Next Steps

• Connection Problems — Broader connectivity troubleshooting
• Certificate Backup — Protect your certificates
• Key Rotation Runbook — Scheduled certificate rotation