Failure Pattern Recognition

Failure Pattern Recognition

Mantis automatically groups recurring deployment failures into patterns — clusters of failures that share the same normalized error signature. Instead of triaging the same broken deployment a dozen times across targets and days, operators see one ranked pattern with its full blast radius (how many deployments and targets it has hit — these are the aggregated counts; environment is recorded per occurrence, not as a pattern-level total) and, optionally, an AI-generated cause and remediation.

Detection is on by default. It runs as a background service inside Mandible and requires no external dependencies for the deterministic grouping; the optional LLM labeling reuses your existing AI Analysis provider configuration.

How it works

1. Fingerprinting. Each failed command result’s error text is normalized (variable tokens such as numbers, UUIDs, paths, and timestamps are masked) and hashed, scoped by an exit-code class. Failures with the same fingerprint belong to the same pattern.
2. Clustering batch. A periodic background pass attributes new un-clustered failures to patterns, updating idempotent counts: distinct deployments, targets, and total occurrences. The “across deployments” count is the primary recurrence signal.
3. Regression. A resolved pattern automatically reopens to active when a new occurrence lands, so a failure you thought was fixed resurfaces instead of silently re-accumulating.
4. Alerts. When a pattern crosses the configured thresholds within the alert window, an edge-triggered pattern_emerging notification fires once (not on every occurrence).
5. LLM labeling (optional). If enabled, the configured AI provider produces a human-readable title, likely cause, and remediation for each cluster. Input is strictly bounded so a pathological log can never blow the provider’s context window.

Enabling and configuring

All knobs live under AI Analysis → Settings in Lens (or via PUT /api/v1/admin/analysis/settings). They require the analysis:manage permission.

Setting	Default	Meaning
pattern_detection_enabled	true	Master toggle for clustering.
pattern_window_days	30	Trailing window of failures considered for clustering.
pattern_batch_interval_secs	300	Seconds between background batch passes.
pattern_retention_days	90	Days a resolved/untouched pattern and its occurrences are kept.
pattern_llm_labeling_enabled	true	Label clusters with the AI provider. Only takes effect when AI Analysis is enabled (off by default), so it incurs no cost until you configure a provider.
pattern_alert_enabled	false	Whether emerging-pattern alerts fire. Off by default — enable it once your thresholds are tuned.
pattern_alert_min_deployments	2	Min distinct deployments before a pattern can alert.
pattern_alert_min_occurrences	5	Min total occurrences before a pattern can alert.
pattern_alert_min_targets	1	Min distinct targets before a pattern can alert.
pattern_alert_window_days	7	Recurrence window the alert thresholds are evaluated over.
pattern_label_max_samples	3	Max error samples sent to the LLM per cluster label.
pattern_label_max_input_chars	4000	Max characters of LLM-label input (context + samples).
pattern_relabel_factor	2	Growth factor (×) before a labeled pattern is relabeled.
pattern_max_batch_size	500	Max failures clustered per batch pass.

Invariants

• Keep pattern_retention_days ≥ pattern_window_days. Retention shorter than the window would prune patterns that are still inside the detection window.
• pattern_alert_window_days vs pattern_window_days is a tuning preference. Occurrences persist until pattern_retention_days, and the alert query is not bounded by the detection window, so a larger alert window does see more data — choose it to match how far back an alert should look, not because a larger value is ignored.

Cost note

Deterministic grouping is cheap and always safe to leave on. LLM labeling makes a provider call per new or significantly-grown cluster and is therefore the only part with a per-use cost. It is on by default but only runs when AI Analysis is enabled (which is off by default), so labeling incurs no cost until you configure an LLM provider. The pattern_relabel_factor controls how much a pattern must grow before it is re-labeled (higher = fewer calls), and pattern_label_max_* bound the per-call input size.

Operating

• Recompute on demand. The Patterns view has a Recompute now button; it is tenant-scoped to the caller and runs on the background service (it does not block the request). The recompute queue is bounded — if it is saturated the request is rejected with 429.
• Mute / resolve. Muting a pattern hides it from the default list and stops it alerting; it never auto-reopens. Resolving closes it but allows regression on a new occurrence.

Monitoring

• The metric mantis_pattern_overload_dropped_total increments when deployment-linked failures age out of the accelerator window before they could be clustered. A rising value means the batch is not keeping up — raise pattern_max_batch_size (re-read each tick). Note pattern_batch_interval_secs is read once at startup, so changing it requires a Mandible restart, not a live reload.

Permissions

Permission	Grants
patterns:read	View patterns, cluster detail, and dashboard stats.
patterns:manage	Recompute, mute, resolve, and reactivate patterns.

Seeded roles: admin and tenant_admin hold both; operator holds patterns:read only. Tenant users see only their own tenant’s patterns; global/system patterns are visible to global admins only.