Skip to content
MantisDocs

Competing Consumers

Mantis has two independent scaling tiers. Mandible replicas compete to drain the dispatch queue, and Thorax instances provide execution capacity that Mandible dispatches onto. This page covers the first; see Multi-Thorax Deployment for the second.

How work is claimed

Section titled “How work is claimed”

Deployment dispatch requests are written to the pending_dispatch_requests table rather than handed straight to a server. This is a PostgreSQL queue, not RabbitMQ, and it is owned by Mandible — Thorax has no database connection at all.

Every Mandible replica polls that table on a 30 second interval and loads the same pending rows. Claiming is a conditional UPDATE from pending to processing: only the replica whose update still sees pending wins, so a queued deployment is dispatched exactly once no matter how many replicas race for it. A replica that loses the race simply moves on.

Recovering abandoned work

Section titled “Recovering abandoned work”

A replica can die between claiming a request and recording its outcome. A row left in processing for longer than the reclaim window is treated as orphaned and reclaimed to pending for another replica to take. The window is derived from grpc.request_timeout_secs and grpc.connect_timeout_secs, as max(1800, 3 × (request + connect) + 600) seconds, so it stays above a dispatch attempt still in flight: reclaiming one sends the deployment out twice. At the defaults (300 and 5) it is 1800 seconds.

Thorax failure is handled separately: the target is reassigned (see Instance Affinity) and the deployment is marked failed rather than retried.

Retries and expiry

Section titled “Retries and expiry”

Failed dispatches are retried up to 10 times. A request that cannot be delivered within 24 hours expires rather than being retried forever, so a permanently unreachable target cannot accumulate work indefinitely.

Adding capacity

Section titled “Adding capacity”

Start another Mandible replica against the same database. It begins polling the queue on its own 30 second tick and racing for rows. There is no shard map to update and no rebalancing step.

Two properties follow:

  • Dispatch throughput scales with replicas, because each claims independently
  • Losing one replica costs only its in-flight claims, which are reclaimed after the 1800 second processing window
Section titled “Related”