Skip to content

Thorax Configuration

Configure the Thorax gRPC orchestration server.

THORAX_CONFIG names the config file outright: when it is set, that file is the only one read, and Thorax fails to start if it is missing.

Otherwise Thorax layers whichever of these exist, each overriding the one before:

  1. thorax.toml in the current directory
  2. ~/.config/mantis/thorax.toml (platform-dependent config directory)
  3. /etc/mantis/thorax.toml

These are layers, not a first-match search: a key set in /etc/mantis/thorax.toml wins over the same key in the user config directory or the current directory, so a file an unprivileged user can write cannot change what the deployed service does. If no file is found, defaults are used. THORAX__* environment variables override file settings; the shared MANTIS_* keys apply only where the file left the field at its default.

name = "thorax-prod-01"
log_level = "info"
FieldTypeDefaultRequiredDescription
namestringhostname-derived (e.g., "thorax-<hostname>")NoServer name for logging. Env: THORAX__NAME
log_levelstring"info"NoLog level: trace, debug, info, warn, error. Env: MANTIS_LOG_LEVEL
retry_timed_out_commandsbooleanfalseNoRe-dispatch a command whose dispatch timed out. Env: THORAX__RETRY_TIMED_OUT_COMMANDS

A dispatch timeout is the command’s own execution budget expiring, so the agent received the command and ran past its deadline. Retrying then hands the same command to an agent that is probably still executing it, which for a deployment step means the step runs twice. Turn retry_timed_out_commands on only where every command you dispatch is idempotent.

TLS configuration. TLS is always enabled and client certificates are always required (mTLS is mandatory).

[tls]
auth_mode = "thumbprint"
cert_path = "/etc/mantis/certs/server.crt"
key_path = "/etc/mantis/certs/server.key"
ca_cert_path = "/etc/mantis/certs/ca.crt"
FieldTypeDefaultRequiredDescription
auth_modestring"thumbprint"NoAuthentication mode: thumbprint (self-signed OK) or ca_signed (CA-signed required). Env: MANTIS_TLS_AUTH_MODE
cert_pathstring-YesServer certificate PEM file. Env: MANTIS_TLS_CERT_PATH
key_pathstring-YesServer private key PEM file. Env: MANTIS_TLS_KEY_PATH
ca_cert_pathstring-Yes (ca_signed mode)CA certificate for client verification. Env: MANTIS_TLS_CA_CERT_PATH
crl_urlstring-NoURL of a published CRL bundle. ca_signed only: fetched before the listener binds, then re-fetched on an interval. If refreshes keep failing until the fetched lists pass nextUpdate, CA-signed clients are refused. Env: MANTIS_TLS_CRL_URL
crl_pathstring-NoA PEM file, or a directory of PEM files, of certificate revocation lists. ca_signed mode only; revocation checking is off while unset. Reloaded without a restart. Env: MANTIS_TLS_CRL_PATH

gRPC server settings.

[grpc]
address = "0.0.0.0"
port = 50051
max_message_size = 16777216
request_timeout_secs = 300
FieldTypeDefaultRequiredDescription
addressstring"0.0.0.0"NoBind address. Env: THORAX__GRPC__ADDRESS
portinteger50051NoListen port. Env: THORAX__GRPC__PORT
max_message_sizeinteger16777216 (16 MB)NoMaximum gRPC message size in bytes
request_timeout_secsinteger300NoRequest timeout (5 minutes)
enrollment_portintegerport + 2 (ca_signed mode)NoCSR-enrollment listener port. Env: THORAX__GRPC__ENROLLMENT_PORT
advertise_addressstringthe bind addressNoAddress Thorax advertises to Mandible for inbound (listen-mode) connections. Unset falls back to address, with 0.0.0.0 reported as 127.0.0.1 — set this explicitly on any multi-host deployment. Env: THORAX__GRPC__ADVERTISE_ADDRESS

Client session management configuration.

[session]
heartbeat_timeout_secs = 90
stale_timeout_secs = 300
cleanup_interval_secs = 60
FieldTypeDefaultRequiredDescription
heartbeat_timeout_secsinteger90NoMark client offline if no heartbeat (3x heartbeat interval)
stale_timeout_secsinteger300NoTime stale before marking offline (5 minutes)
cleanup_interval_secsinteger60NoSession cleanup check interval

Client poll interval, max concurrent commands, and default command timeout are sent to agents by Mandible at registration time — see MANDIBLE__CLIENT_SESSION__* in the Mandible Configuration reference.

Per-instance runtime overrides (live, no restart)

Section titled “Per-instance runtime overrides (live, no restart)”

The session timeout above is seeded from this file at startup. The heartbeat interval is not — it starts from a hard-coded default of 30s, not any config field. Both can be overridden per Thorax instance at runtime without restarting the process. Mandible stores the override and pushes it to the instance on its next heartbeat; the instance applies it live (the heartbeat loop rebuilds its timer; the session-cleanup loop picks up the new timeout).

Set or clear overrides with the admin API (thorax_instances:update permission):

Terminal window
# Set both overrides for an instance
curl -X PATCH .../api/v1/thorax-instances/<id>/config \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"heartbeat_interval_secs": 15, "session_timeout_secs": 600}'
# Clear the heartbeat override (revert to the instance default); omit a field to
# leave it unchanged
curl -X PATCH .../api/v1/thorax-instances/<id>/config \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"heartbeat_interval_secs": null}'

Each field is tri-state: omit to leave unchanged, send null to clear (revert to the configured default), or send a positive integer to set. A session_timeout_secs override maps to the stale→offline threshold (stale_timeout_secs).

Counts failed TLS handshakes per source IP. Crossing failure_threshold within window_secs raises a warning log line and the mantis_tls_rate_limit_alerts_total metric, which is how a misconfigured client, an expired certificate or a scan becomes visible — and, while enforce is on, refuses that address until it has been quiet for one window.

A refusal is decided at accept, before any handshake begins, and increments mantis_tls_connections_refused_total. It is never permanent: a successful handshake clears the address immediately, and an address that simply stops failing is admitted again after window_secs.

[tls_rate_limit]
enabled = true
enforce = true
failure_threshold = 5
window_secs = 60
cleanup_interval_secs = 30
entry_ttl_secs = 300
alert_cooldown_secs = 60
FieldTypeDefaultRequiredDescription
enabledbooleantrueNoEnable TLS handshake failure limiting
enforcebooleantrueNoRefuse an address over the threshold; off = alert only
failure_thresholdinteger5NoFailures within the window before alerting/refusing
window_secsinteger60NoSliding window for rate calculation
cleanup_interval_secsinteger30NoCleanup interval for stale IP entries
entry_ttl_secsinteger300NoTTL for IP entries after last failure (5 minutes)
alert_cooldown_secsinteger60NoCooldown before re-alerting for same IP

Tokio-console settings for async runtime debugging. Only active when the tokio-console feature is enabled.

[console]
enabled = true
address = "127.0.0.1"
port = 6669
retention_secs = 60
publish_interval_ms = 1000
FieldTypeDefaultRequiredDescription
enabledbooleanfalseNoEnable the tokio-console server. Env: THORAX__CONSOLE__ENABLED
addressstring"127.0.0.1"NoBind address. Env: THORAX__CONSOLE__ADDRESS
portinteger0 (sentinel)NoConsole port. Thorax uses 6669 by convention and has no compiled-in default: the serde default is the sentinel 0, which Thorax refuses to start on once enabled is set, so an explicit value is required. Env: THORAX__CONSOLE__PORT
retention_secsinteger60NoHow long to retain completed task data. Env: THORAX__CONSOLE__RETENTION_SECS
publish_interval_msinteger1000NoUpdate interval for connected clients (ms). Env: THORAX__CONSOLE__PUBLISH_INTERVAL_MS
recording_pathstring-NoPath to record console data for offline analysis. Env: THORAX__CONSOLE__RECORDING_PATH

Mandible gRPC client configuration for Thorax-to-Mandible communication. Uses mTLS.

[mandible]
endpoint = "https://mandible:50052"
connect_timeout_secs = 10
request_timeout_secs = 30
tls_cert_path = "/etc/mantis/certs/client.crt"
tls_key_path = "/etc/mantis/certs/client.key"
tls_ca_cert_path = "/etc/mantis/certs/ca.crt"
FieldTypeDefaultRequiredDescription
endpointstring"https://localhost:50052"YesMandible internal gRPC endpoint (must use HTTPS). Env: THORAX__MANDIBLE__ENDPOINT
connect_timeout_secsinteger10NoConnection timeout. Env: THORAX__MANDIBLE__CONNECT_TIMEOUT_SECS
request_timeout_secsinteger30NoRequest timeout. Env: THORAX__MANDIBLE__REQUEST_TIMEOUT_SECS
instance_namestringhostname-derivedNoInstance name for registration; Mandible gives a name to one process at a time
tls_cert_pathstring-YesClient certificate for mTLS. Env: THORAX__MANDIBLE__TLS_CERT_PATH
tls_key_pathstring-YesClient private key for mTLS. Env: THORAX__MANDIBLE__TLS_KEY_PATH
tls_ca_cert_pathstring-YesCA certificate for server verification. Env: THORAX__MANDIBLE__TLS_CA_CERT_PATH

RabbitMQ queue settings for command consumption. TLS is mandatory.

[queue]
enabled = true
host = "rabbitmq.internal"
port = 5671
username = "mantis"
password = "secret"
vhost = "/mantis"
ca_cert_path = "/etc/mantis/certs/rabbitmq-ca.crt"
client_cert_path = "/etc/mantis/certs/rabbitmq-client.crt"
client_key_path = "/etc/mantis/certs/rabbitmq-client.key"
prefetch_count = 10
fallback_on_failure = true
reconnect_delay_secs = 5
FieldTypeDefaultRequiredDescription
enabledbooleanfalseNoEnable RabbitMQ consumption. Env: THORAX__QUEUE__ENABLED. Note: RABBITMQ_ENABLED=true automatically enables cluster coordination (the [cluster] section’s enabled flag) — it does not set this queue flag.
hoststring"localhost"NoRabbitMQ host. Env: THORAX__QUEUE__HOST
portinteger5671NoAMQPS port (TLS only). Env: THORAX__QUEUE__PORT
usernamestring"" (empty — must be set when enabled)NoRabbitMQ username. Validation rejects an empty value when enabled = true. Env: THORAX__QUEUE__USERNAME
passwordstring"" (empty — must be set when enabled)NoRabbitMQ password. Env: THORAX__QUEUE__PASSWORD
vhoststring"/"NoVirtual host. Env: THORAX__QUEUE__VHOST
ca_cert_pathstring-Yes (if enabled)CA certificate for TLS. Env: THORAX__QUEUE__CA_CERT_PATH
client_cert_pathstring-In practiceClient certificate for mTLS. Required against the shipped broker, which sets ssl_options.fail_if_no_peer_cert = true. Env: THORAX__QUEUE__CLIENT_CERT_PATH
client_key_pathstring-In practiceClient private key for mTLS. Required alongside client_cert_path. Env: THORAX__QUEUE__CLIENT_KEY_PATH
prefetch_countinteger10NoConsumer prefetch count
fallback_on_failurebooleantrueNoFall back to polling when RabbitMQ is unavailable. Env: THORAX__QUEUE__FALLBACK_ON_FAILURE
reconnect_delay_secsinteger5NoReconnection delay after failure, and the interval between Thorax’s queue health probes. Env: THORAX__QUEUE__RECONNECT_DELAY_SECS
publisher_confirmsbooleantrueNoRequire broker publisher confirmations
health_check_interval_secsinteger10NoRead by Mandible only. Thorax paces its health probes with reconnect_delay_secs, so setting this here changes nothing

WireGuard overlay-controller settings. Off by default — when enabled = false Thorax runs with no overlay and this section has no effect. When enabled, Thorax acts as the overlay controller: it allocates overlay IPs to agents and reports its own controller_overlay_ip to Mandible.

[wireguard]
enabled = true
overlay_ip = "10.99.0.1"
overlay_prefix_len = 16
udp_bind_addr = "0.0.0.0:51820"
udp_advertise_addr = "thorax.example.com:51820"
relay_url = "wss://relay.example.com/v1/tunnel"
FieldTypeDefaultRequiredDescription
enabledbooleanfalseNoMaster switch for the overlay. Env: THORAX__WIREGUARD__ENABLED
overlay_ipstring10.99.0.1NoController’s address on the overlay; reported to Mandible as controller_overlay_ip. Env: THORAX__WIREGUARD__OVERLAY_IP
overlay_prefix_leninteger16NoOverlay subnet prefix length (e.g. 16 for 10.99.0.0/16). Env: THORAX__WIREGUARD__OVERLAY_PREFIX_LEN
udp_bind_addrstring0.0.0.0:51820Nohost:port the WireGuard UDP data-plane socket binds to. Env: THORAX__WIREGUARD__UDP_BIND_ADDR
udp_advertise_addrstringfalls back to udp_bind_addrNoPublicly reachable host:port advertised to agents. Env: THORAX__WIREGUARD__UDP_ADVERTISE_ADDR
private_key_pathstring/var/lib/mantis/thorax-wireguard.keyNoPath where the controller’s WireGuard (X25519) keypair is persisted across restarts. Env: THORAX__WIREGUARD__PRIVATE_KEY_PATH
signing_private_key_pathstringprivate_key_path with its extension replaced by .signing (/var/lib/mantis/thorax-wireguard.signing)NoPath for the controller’s Ed25519 relay signing key. Env: THORAX__WIREGUARD__SIGNING_PRIVATE_KEY_PATH
relay_urlstring-NoWebSocket relay URL for NAT-fallback frame exchange; unset = direct-UDP only. Env: THORAX__WIREGUARD__RELAY_URL
relay_expected_identitystring-NoRelay identity to pin; unset leaves the relay authenticated by TLS alone. Env: THORAX__WIREGUARD__RELAY_EXPECTED_IDENTITY

Cluster coordination for multi-Thorax deployments via Redis.

[cluster]
enabled = true
instance_id = "thorax-prod-01"
redis_url = "redis://redis:6379"
heartbeat_interval_secs = 10
liveness_timeout_secs = 30
namespace = "thorax"
FieldTypeDefaultRequiredDescription
enabledbooleanfalseNoEnable cluster coordination. Automatically enabled when RABBITMQ_ENABLED=true. Env: THORAX__CLUSTER__ENABLED, RABBITMQ_ENABLED
instance_idstringhostname-derivedNoName this Thorax registers under. Redis keys use the ID Mandible assigns in return; the certificate CN is recorded separately and a fleet may share one. Env: THORAX__CLUSTER__INSTANCE_ID
redis_urlstring"redis://localhost:6379"NoRedis connection URL. Env: THORAX__CLUSTER__REDIS_URL
heartbeat_interval_secsinteger10NoHeartbeat interval to Redis. Env: THORAX__CLUSTER__HEARTBEAT_INTERVAL_SECS
liveness_timeout_secsinteger30NoTimeout before considering instance dead. Env: THORAX__CLUSTER__LIVENESS_TIMEOUT_SECS
namespacestring"thorax"NoRedis key namespace prefix. Env: THORAX__CLUSTER__NAMESPACE
require_hardened_redisbooleanfalseNoRefuse to start unless redis_url carries TLS and credentials. Env: THORAX__CLUSTER__REQUIRE_HARDENED_REDIS

The metrics, health and WireGuard-relay HTTP listener.

[metrics]
bind_addr = "127.0.0.1:9090"
auth_token = "<openssl rand -hex 32>"
[metrics.tls]
cert_path = "/etc/mantis/certs/thorax-cert.pem"
key_path = "/etc/mantis/certs/thorax-key.pem"
FieldTypeDefaultRequiredDescription
bind_addrstring127.0.0.1:9090Nohost:port the listener binds to; empty disables it, and /health and the relay with it. Env: THORAX__METRICS__BIND_ADDR
auth_tokenstring-When bind_addr is not loopbackBearer token /metrics and /health require in Authorization; at least 32 visible ASCII characters. Env: THORAX__METRICS__AUTH_TOKEN
tls.cert_pathstring-With tlsCertificate the listener serves. Env: THORAX__METRICS__TLS__CERT_PATH
tls.key_pathstring-With tlsIts private key. Env: THORAX__METRICS__TLS__KEY_PATH

/health discloses cluster topology, so both endpoints answer 401 without the token once one is set, and Thorax refuses to start with a bind beyond loopback and no token. The WireGuard relay WebSocket shares this listener but authenticates its agents itself and does not take the token, so a Thorax serving relay agents widens bind_addr to an address they can reach.

The listener is plaintext HTTP unless [metrics.tls] is set. Set it, put a TLS-terminating proxy in front, or keep the listener on a management interface: over plaintext the token and the relay’s handshake cross the network in the clear. With it set the relay is reached as wss://, thorax --health-check verifies the listener by its configured certificate, and mantisctl probes it over HTTPS once given the certificate’s CA as thorax_metrics_ca_cert_path. A client CA (tls.client_ca_cert_path), which Mandible’s listener accepts, is refused here: relay agents and the health check present no certificate.

Off-target steps, which this instance runs itself rather than through an agent.

[execution]
inherit_env = ["JAVA_HOME"]
FieldTypeDefaultRequiredDescription
inherit_envarray[]NoHost environment variables an off-target step inherits besides the base set. Env: THORAX__EXECUTION__INHERIT_ENV

A step runs with only the base set of this process’s environment (PATH, HOME, the locale and the temporary directory, and their Windows equivalents) plus its own variables, so Thorax’s keys and connection strings never reach tenant code. inherit_env refuses them too: Mantis’s own variables (MANTIS_*, MANDIBLE__*, THORAX__*, TARSUS__* and DATABASE_URL) stop Thorax at startup. Its process tree is stopped as a whole at a timeout or a cancellation.

The environment layer is applied by the shared instinct::config loader, which builds it from the THORAX prefix with __ as the separator — so every field is settable as THORAX__<SECTION>__<FIELD>. The cross-cutting MANTIS_* keys carry no service prefix and are applied separately, after the prefixed layer, so a THORAX__* value the operator set wins over the shared one. See each section above for the specific env var names.

Common env vars:

Environment VariableDescription
THORAX_CONFIGPath to config file
THORAX__NAMEServer name
MANTIS_LOG_LEVELLog level
THORAX__MANDIBLE__ENDPOINTMandible gRPC endpoint (full HTTPS URL, e.g. https://mandible:50052)
THORAX__GRPC__ADDRESSgRPC bind address
THORAX__GRPC__PORTgRPC port
THORAX__GRPC__ENROLLMENT_PORTCSR enrollment listener port
THORAX__GRPC__ADVERTISE_ADDRESSAddress advertised to Mandible for inbound connections
THORAX__METRICS__BIND_ADDRPrometheus metrics bind address (default 127.0.0.1:9090)
THORAX__METRICS__AUTH_TOKENBearer token /metrics and /health require
THORAX__METRICS__TLS__CERT_PATH, THORAX__METRICS__TLS__KEY_PATHCertificate and key the metrics listener serves over TLS (unset = plaintext)
MANTIS_TLS_CERT_PATHServer TLS certificate
MANTIS_TLS_KEY_PATHServer TLS private key
MANTIS_TLS_CA_CERT_PATHCA certificate
MANTIS_TLS_AUTH_MODETLS auth mode (thumbprint or ca_signed)
RABBITMQ_ENABLEDEnable RabbitMQ + cluster mode

See Environment Variables for complete reference.

thorax.toml
name = "thorax-prod-01"
log_level = "info"
[tls]
auth_mode = "ca_signed"
cert_path = "/etc/mantis/certs/server.crt"
key_path = "/etc/mantis/certs/server.key"
ca_cert_path = "/etc/mantis/certs/ca.crt"
[grpc]
address = "0.0.0.0"
port = 50051
[session]
heartbeat_timeout_secs = 90
# max_concurrent_commands and default_command_timeout_secs are sent by Mandible
# at registration time — configure them via MANDIBLE__CLIENT_SESSION__* instead.
[mandible]
endpoint = "https://mandible.internal:50052"
tls_cert_path = "/etc/mantis/certs/client.crt"
tls_key_path = "/etc/mantis/certs/client.key"
tls_ca_cert_path = "/etc/mantis/certs/ca.crt"
[queue]
enabled = true
host = "rabbitmq.internal"
port = 5671
username = "mantis"
password = "secret"
vhost = "/mantis"
ca_cert_path = "/etc/mantis/certs/rabbitmq-ca.crt"
client_cert_path = "/etc/mantis/certs/rabbitmq-client.crt"
client_key_path = "/etc/mantis/certs/rabbitmq-client.key"
[cluster]
enabled = true
redis_url = "redis://redis.internal:6379"
namespace = "mantis-prod"