Health Status

Understand and monitor the health status of your targets.

Health States

Targets can be in one of three health states:

State	Icon	Description
Online	●	Target is connected and responding
Stale	◐	Target missed recent heartbeat
Offline	○	Target is not connected

Health State Diagram

Online State

A target is Online when:

Tarsus agent is running
The agent’s connection to Mandible is active
Heartbeats are received by Mandible within threshold

┌─────────────────────────────────────────────────────────────┐
│  Target: web-prod-01                                        │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Status:     ● Online                                       │
│  Last Seen:  Just now                                       │
│  Mode:       Listen                                         │
│  (No Uptime field — uptime is shown only on Thorax          │
│   instances, not on targets.)                               │
│                                                             │
│  ✓ Ready to receive deployments                             │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Online Indicators

Indicator	Value
Last seen	Within heartbeat interval
Connection	Active
Deployable	Yes

Stale State

A target becomes Stale when:

Heartbeat not received within threshold (default: 90 seconds)
Connection may be intermittent
Target might be under heavy load

┌─────────────────────────────────────────────────────────────┐
│  Target: web-prod-03                                        │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Status:     ◐ Stale                                        │
│  Last Seen:  5 minutes ago                                  │
│  Mode:       Listen                                         │
│                                                             │
│  ⚠ Target may be experiencing issues                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Stale Indicators

Indicator	Value
Last seen	> heartbeat threshold
Connection	Unknown/Intermittent
Deployable	Yes (status is not enforced)

Common Stale Causes

Cause	Resolution
Network congestion	Wait for recovery
High CPU load	Investigate target performance
Tarsus overloaded	Check Tarsus logs
Firewall issues	Verify connectivity

Offline State

A target is Offline when:

No heartbeat for an extended period (~390s total: 90s to Stale, then a 300s stale dwell before Offline; Mandible independently forces Offline at 360s since the last heartbeat)
Connection is confirmed lost
Tarsus agent may not be running

┌─────────────────────────────────────────────────────────────┐
│  Target: worker-01                                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Status:     ○ Offline                                      │
│  Last Seen:  2 hours ago                                    │
│  Mode:       Listen                                         │
│                                                             │
│  ✗ Target is not available for deployments                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Offline Indicators

Indicator	Value
Last seen	> offline threshold
Connection	Disconnected
Deployable	Yes (status is not enforced)

Common Offline Causes

Cause	Resolution
Server powered off	Power on the server
Tarsus not running	Start Tarsus service
Network unreachable	Fix network connectivity
Certificate expired	Renew certificates

Health Monitoring

Dashboard Overview

┌─────────────────────────────────────────────────────────────┐
│  Target Health Summary                                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │   Online    │  │    Stale    │  │   Offline   │         │
│  │     10      │  │      1      │  │      1      │         │
│  │    (83%)    │  │    (8%)     │  │    (8%)     │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
│                                                             │
│  Total Targets: 12                                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Health Alerts

Per-target health-alert configuration (notification rules that fire when a specific target changes state) is not yet available (planned — not yet available). Today you monitor health state from the dashboard and target detail views.

Health History

Mantis does not retain a per-target history of state transitions. The target record keeps only the current status plus last_seen_at and updated_at — there is no status-history table or timeline view. Use notifications (below) or your own monitoring to record transitions over time.

Heartbeat Configuration

In both Listen and Poll mode, the Tarsus agent sends heartbeats directly to Mandible, which tracks target liveness and updates the health status. The connection mode only affects how commands are delivered for execution, not how health is tracked.

The stale and offline thresholds below are configured server-side in Thorax’s [session] block (thorax.toml); Tarsus itself does not expose stale/offline threshold settings.

Listen Mode Heartbeat

In Listen mode, Tarsus sends heartbeats to Mandible while a Thorax instance connects inbound to deliver commands:

Setting	Default	Description
Heartbeat interval	30 seconds	How often Tarsus reports to Mandible
Stale threshold	90 seconds	When to mark stale (`heartbeat_timeout_secs`)
Offline threshold	5 minutes	When to mark offline (`stale_timeout_secs`)

Poll Mode Heartbeat

In Poll mode, Tarsus also heartbeats directly to Mandible, separately from polling its assigned Thorax for commands:

Setting	Default	Description
Heartbeat interval	30 seconds	How often Tarsus reports to Mandible
Stale threshold	90 seconds	When to mark stale (`heartbeat_timeout_secs`)
Offline threshold	5 minutes	When to mark offline (`stale_timeout_secs`)

Health Checks

Built-in Health Checks

Liveness is derived solely from heartbeat recency — Mantis marks a target Stale, then Offline, based on how long ago the last heartbeat arrived. There is no TCP connectivity probe and no response-time measurement or storage. The only other automatic check is per-connection mTLS certificate validation.

Check	Description	Frequency
Heartbeat	Time since last heartbeat vs. threshold	Per heartbeat
TLS	Certificate validation	Per connection

Custom Health Checks

Define custom health checks in actions:

#!/bin/bash
# Check application is running
if ! pgrep -x "myapp" > /dev/null; then
    echo "Application not running"
    exit 1
fi

# Check port is listening
if ! nc -z localhost 8080; then
    echo "Port 8080 not listening"
    exit 1
fi

# Check disk space
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$DISK_USAGE" -gt 90 ]; then
    echo "Disk usage critical: ${DISK_USAGE}%"
    exit 1
fi

echo "All health checks passed"
exit 0

Health and Deployment

Health status does not gate deployment. Target selection filters by id, tag and tenant only — a deployment dispatches to its selected targets regardless of whether they are Online, Stale or Offline, and there is no “include by status” or “wait for stale recovery” setting. A command sent to an Offline target simply waits (listen mode) or is picked up when the agent next polls; if the agent never returns, the command times out.

Check target health yourself before deploying — the target picker shows current status passively so you can deselect targets you know are down.

Troubleshooting Health Issues

Target Stuck in Stale

Symptoms:

Target shows Stale for extended periods
Never transitions to Offline or Online

Diagnosis:

Check network latency
Verify Tarsus logs
Check system resources

Resolution:

Restart Tarsus if needed
Fix network issues
Increase heartbeat thresholds if appropriate

Frequent Online/Stale Transitions

Symptoms:

Target flaps between Online and Stale
Health history shows many transitions

Diagnosis:

Check network stability
Look for resource contention
Review system logs

Resolution:

Fix underlying network issues
Increase threshold values
Investigate root cause

Target Won’t Come Online

Symptoms:

Target remains Offline
Tarsus appears to be running

Diagnosis:

Verify Tarsus is listening (Listen mode)
Check firewall rules
Validate certificates
Test network connectivity

Resolution:

# Check Tarsus status
systemctl status tarsus

# Test connectivity (Listen mode)
nc -zv target-hostname 9342

# Check certificates
openssl s_client -connect target-hostname:9342

# View Tarsus logs
journalctl -u tarsus -f

Best Practices

1. Monitor Health Proactively

Review dashboard regularly
Set up alerts for offline targets
Investigate stale targets promptly

2. Right-Size Thresholds

Environment	Stale Threshold	Offline Threshold
Low-latency	60 seconds	3 minutes
Standard	90 seconds	5 minutes
High-latency	5 minutes	15 minutes

3. Document Health Issues

Track and document recurring health issues:

Target	Issue	Frequency	Resolution
web-03	Network flapping	Weekly	ISP issue
db-01	High load stale	Monthly	Scale resources

4. Plan for Offline Targets

Have procedures for offline targets
Don’t deploy to offline targets in production
Investigate offline targets before deploying

Next Steps

Managing Targets - Target management
Listen vs Poll Mode - Connection modes
Tagging - Organize targets
Overview - Return to targets overview