Competing Consumers

Mantis has two independent scaling tiers. Mandible replicas compete to drain the dispatch queue, and Thorax instances provide execution capacity that Mandible dispatches onto. This page covers the first; see Multi-Thorax Deployment for the second.

How work is claimed

Deployment dispatch requests are written to the pending_dispatch_requests table rather than handed straight to a server. This is a PostgreSQL queue, not RabbitMQ, and it is owned by Mandible — Thorax has no database connection at all.

Every Mandible replica polls that table on a 30 second interval and loads the same pending rows. Claiming is a conditional UPDATE from pending to processing: only the replica whose update still sees pending wins, so a queued deployment is dispatched exactly once no matter how many replicas race for it. A replica that loses the race simply moves on.

Recovering abandoned work

A replica can die between claiming a request and recording its outcome. A row left in processing for more than 300 seconds is treated as orphaned and reclaimed to pending for another replica to take.

Thorax failure is handled separately: the target is reassigned (see Instance Affinity) and the deployment is marked failed rather than retried.

Retries and expiry

Failed dispatches are retried up to 10 times. A request that cannot be delivered within 24 hours expires rather than being retried forever, so a permanently unreachable target cannot accumulate work indefinitely.

Adding capacity

Start another Mandible replica against the same database. It begins polling the queue on its own 30 second tick and racing for rows. There is no shard map to update and no rebalancing step.

Two properties follow:

• Dispatch throughput scales with replicas, because each claims independently
• Losing one replica costs only its in-flight claims, which are reclaimed after the 300 second processing window

Related

• Instance Affinity — how targets are pinned to a server and moved when one fails