Skip to content

Incident Management

What you'll learn: How incidents are created, the incident lifecycle, and how to triage and resolve them.

What is an incident?

An incident is a policy violation record. When the engine evaluates a step and one or more policies return warn or block, it reports the violation to the configured incident sink (by default, the same destination as the behavioral trace, see Configuring the incident sink).

Violations are grouped, not duplicated. Each violated policy produces an incident, but while an active or ignored incident already exists for the same agent, policy and environment (at either enforcement point, step_execution or agent_registration), a repeat violation updates it instead of creating another: last_occurrence moves forward, occurrence_count increases, and task_id and details show the latest occurrence. Once an incident is resolved, the next violation opens a new one. A step that breaks three policies still yields three incidents.

Incidents provide the audit trail: they record what was violated, when, by which agent, and in which task.

Incident lifecycle

active  →  resolved
        →  ignored
Status Meaning
active What the platform creates: a violation nobody has dealt with yet.
resolved The violation has been addressed (code fixed, policy adjusted, or root cause mitigated).
ignored Reviewed and determined to be a false positive or acceptable risk.

A resolved or ignored incident can be reactivated. The API also accepts open and fixed as status filter values, but nothing in the platform sets either today: an incident is not moved to fixed when the agent later passes the policy.

Incident data

Each incident includes:

Field Description
incident_id The incident's identifier (inc_...).
agent_id The agent that triggered the violation.
policy_id The policy that was violated (null when the reported policy no longer resolves; the incident keeps a policy_snapshot of its name, manifest, rule and severity). The title reads Policy violation: <policy name> (<manifest name>).
severity The violated policy's severity.
status active, resolved or ignored (see above).
task_id The task in which the violation last occurred (for step violations).
environment The agent's environment.
details enforcement_point (step_execution or agent_registration), step_name, kind, action_id, action (warn or block), risk_score (0.0 to 1.0, for the whole step) and violation_details from the rule.
created_at, last_occurrence, occurrence_count When the incident was created, when the violation last occurred, and how many violations were grouped into it.
resolved_at, resolved_by, resolution_notes Set when the incident is resolved.

Dashboard workflow

  1. Navigate to Incidents in the sidebar.
  2. Filter by status, severity, environment, or the agent's risk classification.
  3. Click a row to expand it: the table shows ID, title, severity, status, environment, agent, occurrence count and last occurrence, and the expanded row shows the full details.
  4. Change its status with the buttons at the foot of the expanded row. Each asks for confirmation first:
  5. Resolve (active incident): the violation has been dealt with. You are recorded in resolved_by, and the next violation of the same policy by the agent opens a new incident.
  6. Ignore (active incident): a false alarm or an accepted risk. Later violations are still counted on this incident, which stays ignored.
  7. Reactivate (resolved or ignored incident): back to active.

Any workspace member who can see the incident can change its status; there is no role restriction. The same changes are available through the API endpoints below, with a member's JWT.

API endpoints

Method Path Description
POST /api/v1/incidents Report violations. Called by the engine with the agent's KvKey- API key; returns 201 with the created and updated counts.
GET /api/v1/incidents List incidents. Filters: agent_id, policy_id, environment, status, severity; skip and limit (default 50, max 1000).
GET /api/v1/incidents/{incident_id} Get incident details. {incident_id} is the inc_... identifier.
PUT /api/v1/incidents/{incident_id}/resolve Resolve incident. Optional body resolution_notes.
PUT /api/v1/incidents/{incident_id}/ignore Ignore incident.
PUT /api/v1/incidents/{incident_id}/reactivate Reactivate a resolved or ignored incident (400 from any other status).

Incidents are never deleted — they are part of the audit trail. They travel with their agent: moving an agent to another workspace carries its logs and incidents along, hash chain intact, and each moved incident carries a self-contained snapshot of the policy it violated, so deleting the workspace it left cannot destroy it.

Configuring the incident sink

Incidents follow the same location + format sink model as behavioral traces. By default the incident sink inherits the trace sink (KV_LOG_LOCATION / KV_LOG_FORMAT), substituting the …/api/v1/incidents path for the Kyvvu kv format. So if traces already flow to the platform, incidents do too — no extra configuration needed.

To route incidents to a different destination, set KV_INCIDENT_LOCATION (and optionally KV_INCIDENT_FORMAT):

# Send incidents to the platform explicitly (kv format → …/api/v1/incidents)
export KV_INCIDENT_LOCATION=https://platform.kyvvu.com
export KV_INCIDENT_FORMAT=kv

KV_INCIDENT_LOCATION accepts the same values as KV_LOG_LOCATION — an HTTP URL, a file path (JSONL), stdout, or none/empty to disable:

# Local debugging — print incident JSON to stdout
export KV_INCIDENT_LOCATION=stdout

# Emit a standalone kyvvu.incident OpenTelemetry span
export KV_INCIDENT_LOCATION=http://collector:4318
export KV_INCIDENT_FORMAT=otlp

Next steps