Incidents & ITSM

Overview

The Incidents & ITSM plugin is an Enterprise plugin that adds full incident lifecycle management to NetStacks: create, track, assign, comment on, and resolve incidents, with optional one- or two-way synchronization to an external IT Service Management platform. Incidents can be created manually or auto-created from alerts produced by the Alert Pipeline plugin.

Key Features

• Incident lifecycle -- five states and five severity levels, with linked alerts and threaded comments
• Auto-creation rules -- match alerts by severity, source, and summary, then create an incident from a title template
• ITSM sync -- push incidents to ServiceNow, Jira, or a generic webhook; pull state changes back from ServiceNow and Jira
• Knowledge indexing -- resolved incidents are auto-indexed to the knowledge base as a reusable resolution narrative

Plugin Settings

The plugin exposes exactly two settings in its manifest settings_schema, configured under /api/admin/plugins/incidents:

Incident Lifecycle & Severity

Incident States

Every incident moves through one of five states. New incidents default to open. Moving an incident to resolved or closed triggers knowledge-base indexing of the resolution.

Incident Severity

Incidents use five severity levels: critical, high, medium, low, and info. The default for new incidents (and the plugin default_severity setting) is medium.

How Sync Works

Sync is driven by a background worker that runs two independent loops. An incident is only synced when it references an ITSM config (itsm_config_id) and that config has sync_enabled = true.

Push loop (NetStacks → ITSM)

• Runs every 30 seconds (fixed worker cadence, PUSH_INTERVAL_SECONDS).
• Selects incidents whose sync_state = pending. An incident becomes pending automatically whenever its state changes (the update endpoint sets sync_state = pending and increments sync_version).
• If the incident has no external_id yet, the provider create_incident is called and the returned external ID is stored. Otherwise update_incident pushes changes.
• On success the row is marked synced; on failure it is marked error and sync_error_message is recorded.

Pull loop (ITSM → NetStacks)

• Runs every 300 seconds (PULL_INTERVAL_SECONDS), after a 60-second startup delay.
• Selects synced incidents that have an external_id and are not yet closed, then calls the provider get_incident.
• If the external state differs, the local state is updated to match. Webhook configs return nothing on pull, so they are push-only.

ITSM Providers

The provider field on an ITSM config selects one of three integrations:

servicenow

Uses the ServiceNow REST Table API (/api/now/table/incident). Full two-way sync: create, update, pull state, and add work notes. Supports basic (username/password) and bearer (token) auth.

jira

Uses Jira REST API v2 (/rest/api/2/issue). Full two-way sync including status transitions and comments. Jira Cloud uses basic auth with an email and API token; Jira Server can use bearer.

webhook

Push-only. POSTs a JSON payload to the configured base_url for each event (incident.created, incident.updated, incident.comment_added). Use this for custom ITSM tools or automation pipelines. Supports bearer and custom header auth. It returns no external state, so it is never pulled.

Severity → priority mapping (in the provider)

Each provider maps incident severity to its own priority scheme. These mappings live in provider code, not in the rules.

State is mapped too: NetStacks open maps to ServiceNow New, in_progress to In Progress, resolved to Resolved, and closed to Closed. For Jira, the provider looks up the matching workflow transition by status category (To Do, In Progress, Done).

The optional field_mappings object adds extra mappings. For Jira, project_key and issue_type are read from field_mappings (defaults are INC and Bug). Any other entry maps a local incident field name to a target ITSM field name.

Auto-Creation Rules

Creation rules turn alerts into incidents automatically. When the Alert Pipeline finishes evaluating an alert, it calls the incidents plugin, which checks the alert against each enabled rule in priority order (lower priority number = evaluated first). The first matching rule wins.

Rule fields

• name -- unique rule name within the org.
• enabled -- whether the rule is evaluated (default true).
• priority -- evaluation order, lower is first (default 100).
• alert_pattern -- match criteria (see below). An empty pattern matches all alerts.
• incident_title_template -- title with {variables} substituted from the alert.
• incident_severity -- the incident severity to assign (critical, high, medium, low, info).
• assign_to_role -- optional role to assign the incident to.
• itsm_config_id and auto_sync_to_itsm -- optionally sync the new incident to ITSM on creation.

alert_pattern matching

The alert_pattern object supports three keys, all optional and AND-combined:

• severity -- threshold; the alert severity must be at least this level. Accepts a single value or a list (the minimum level in the list is used).
• source -- case-insensitive regex matched against the alert source (a list is joined with |).
• summary_pattern -- case-insensitive regex matched against the alert summary.

Title template variables

The title template substitutes these tokens from the alert: {summary}, {source}, {severity}, {device_id}, and {alert_id}.

API & Config Examples

All data routes are reached through the controller proxy at /api/plugins/incidents/* and require an org_id query parameter. Examples use a placeholder controller host and bearer token.

Create a ServiceNow ITSM config

POST /api/plugins/incidents/itsm/configs?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "name": "Prod ServiceNow",
  "provider": "servicenow",
  "base_url": "https://company.service-now.com",
  "auth_config": {
    "type": "basic",
    "credentials": {
      "username": "netstacks-svc",
      "password": "********"
    }
  },
  "sync_enabled": true,
  "sync_interval_minutes": 15,
  "field_mappings": {
    "assigned_to": "assignment_group"
  }
}

Create a Jira ITSM config

POST /api/plugins/incidents/itsm/configs?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "name": "NOC Jira",
  "provider": "jira",
  "base_url": "https://company.atlassian.net",
  "auth_config": {
    "type": "basic",
    "credentials": {
      "username": "[email protected]",
      "password": "YOUR_JIRA_API_TOKEN"
    }
  },
  "sync_enabled": true,
  "field_mappings": {
    "project_key": "NOC",
    "issue_type": "Incident"
  }
}

Create a generic webhook config

POST /api/plugins/incidents/itsm/configs?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "name": "Custom Webhook",
  "provider": "webhook",
  "base_url": "https://hooks.example.com/netstacks/incidents",
  "auth_config": {
    "type": "header",
    "credentials": { "name": "X-API-Key", "value": "********" }
  },
  "sync_enabled": true
}

The webhook receives a JSON envelope per event. Example payload for a created incident:

{
  "event": "incident.created",
  "source": "netstacks-incidents",
  "external_id": "WH-1A2B3C4D",
  "incident": {
    "id": "8f1c...",
    "title": "BGP peer down on core-rtr-01",
    "description": "...",
    "severity": "critical",
    "state": "open",
    "created_at": "2026-06-15T14:31:00Z"
  }
}

Test connectivity

Test an existing config by its config_id. The plugin instantiates the provider and makes a real authenticated call.

curl -X POST \
  "https://netstacks.example.com/api/plugins/incidents/itsm/configs/CONFIG_ID/test-connection?org_id=ORG_UUID" \
  -H "Authorization: Bearer YOUR_TOKEN"

# Response
{
  "success": true,
  "provider": "servicenow",
  "base_url": "https://company.service-now.com",
  "message": "Successfully connected to ServiceNow instance"
}

Create an auto-creation rule

POST /api/plugins/incidents/admin/creation-rules?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "name": "Critical core-router alerts",
  "enabled": true,
  "priority": 10,
  "alert_pattern": {
    "severity": "critical",
    "source": "^core-rtr-",
    "summary_pattern": "bgp|ospf|isis"
  },
  "incident_title_template": "[{severity}] {summary} on {source}",
  "incident_severity": "critical",
  "assign_to_role": "network-ops",
  "itsm_config_id": "CONFIG_ID",
  "auto_sync_to_itsm": true
}

Create an incident directly

POST /api/plugins/incidents/admin/incidents?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "title": "BGP peer 10.0.0.2 down on core-rtr-01",
  "description": "Detected during morning checks",
  "severity": "high",
  "state": "open",
  "itsm_config_id": "CONFIG_ID"
}

Transition an incident state

Updating state marks the incident sync_state = pending so the next push loop syncs it outward. Setting resolved or closed also triggers knowledge-base indexing.

PUT /api/plugins/incidents/admin/incidents/INCIDENT_ID?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{ "state": "resolved" }

Comments & Knowledge Base

Incident comments

Comments are managed under the incident. Adding a comment to an incident that is linked to an ITSM system also pushes the comment outward (ServiceNow work note, Jira comment, or webhook incident.comment_added event).

# Add a comment (X-User-Id header identifies the author)
POST /api/plugins/incidents/admin/incidents/INCIDENT_ID/comments?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
X-User-Id: USER_UUID
Content-Type: application/json

{ "comment_text": "Confirmed adjacency restored after clearing the session." }

# List comments
GET /api/plugins/incidents/admin/incidents/INCIDENT_ID/comments?org_id=ORG_UUID

Linking alerts

Alerts can be linked to and unlinked from an incident. Auto-created incidents link their originating alert automatically; you can add more correlated alerts via the link endpoint.

POST /api/plugins/incidents/admin/incidents/INCIDENT_ID/alerts?org_id=ORG_UUID
Authorization: Bearer YOUR_TOKEN
X-User-Id: USER_UUID
Content-Type: application/json

{ "alert_id": "ALERT_UUID" }

Knowledge-base indexing

When an incident transitions to resolved or closed, the plugin builds a markdown resolution narrative and indexes it to the NetStacks knowledge base (source type incident, category incident_resolution). The narrative pulls in the incident description, root cause and resolution from linked alerts, and the investigation timeline from alert triage events, plus searchable tags. This makes past incident resolutions retrievable when triaging future alerts.

Terminal Integration

The plugin contributes an Incidents panel to the NetStacks Terminal activity bar (icon AlertTriangle), backed by the /admin/incidents data endpoint and refreshing every 30 seconds. The list shows title, severity, state, linked alert count, the external ITSM ID, and creation time.

Incident detail

Click a row to open the incident. From the detail view you can edit the title, description, severity, state, and assignment, link or unlink alerts, and add comments. Changing the state drives the same sync and knowledge-indexing behavior as the API.

Filtering

The list endpoint accepts state, severity, and assigned_to filters, so the panel can be narrowed to, for example, open critical incidents assigned to you.

Q&A

Q: Which ITSM platforms are supported?

A: ServiceNow (REST Table API) and Jira (REST API v2) with full two-way sync, plus a generic webhook provider that is push-only. Webhook is ideal for custom ITSM tools or automation pipelines.

Q: What are the incident states and severities?

A: States are open, acknowledged, in_progress, resolved, and closed (default open). Severities are critical, high, medium, low, and info (default medium).

Q: How often does sync run?

A: The push loop (NetStacks → ITSM) runs every 30 seconds and the pull loop (ITSM → NetStacks) every 300 seconds. These are fixed worker cadences. The sync_enabled flag on each ITSM config turns syncing on or off for that config.

Q: How does severity map to ITSM priority?

A: The provider performs the mapping. ServiceNow uses impact and urgency (critical = 1/1), Jira uses named priorities (critical = Highest, high = High, medium = Medium, low = Low, info = Lowest). Creation rules set the incident severity; they do not set ITSM priority directly.

Q: Can I control which alerts create incidents?

A: Yes, via creation rules. Each rule has an alert_pattern matching on severity threshold, source regex, and summary regex, and sets the resulting incident severity and (optionally) an ITSM config to auto-sync to. The first matching rule by ascending priority wins.

Q: Are duplicate incidents prevented?

A: Yes. Before auto-creating, the plugin checks whether the alert is already linked to a non-resolved, non-closed incident and links to it instead of creating a new one.

Q: What happens to credentials in API responses?

A: The credentials inside an ITSM config auth_config are returned as ***MASKED*** on all reads. Supply real credentials only on create/update.

Q: What happens when an incident is resolved?

A: Setting state to resolved or closed pushes the state to the ITSM system on the next push loop and indexes a resolution narrative (with root cause, resolution, and triage timeline from linked alerts) to the knowledge base.

Troubleshooting

Connection test fails

• Run the test-connection endpoint for the config and read the returned message -- it reports the exact cause (bad URL, HTTP 401 auth failure, HTTP 403 permission, or timeout).
• Verify base_url is reachable from the plugin container and that auth_config.type matches the credentials you provided (basic vs bearer).

Incident stuck in error / not syncing

• Check the incident's sync_state and sync_error_message fields. An error state records the provider error (truncated to 500 chars).
• Confirm the linked ITSM config has sync_enabled = true. Only enabled configs are processed by the push and pull loops.
• Remember the push loop runs every 30s and pull every 300s; allow a cycle before assuming a failure.

Pull sync never updates state

• The webhook provider is push-only and returns no state, so it is never pulled. Use ServiceNow or Jira for two-way sync.
• Closed incidents are excluded from the pull loop by design.

Jira state transitions do not apply

• Jira transitions depend on the project workflow. The provider looks up an available transition whose target status matches the mapped category (To Do, In Progress, Done). If your workflow has no matching transition, the status will not change and a warning is logged.

Cannot delete an ITSM config

• Deletion is rejected if any incident still references the config. Reassign or delete those incidents first.

Field mapping not applied

• In field_mappings, keys are local incident field names and values are target ITSM field names. For Jira, set project_key and issue_type here -- if omitted they default to INC and Bug.

Related Features

• Plugin System -- how Enterprise plugins are packaged, proxied, and managed
• Alert Pipeline -- the alert source that feeds auto-creation rules
• Knowledge Base -- where resolved incident narratives are indexed for reuse
• Roles & Permissions -- roles referenced by assign_to_role on creation rules
• System Settings -- plugin enablement and system-wide configuration