Alert Pipeline

Overview

The Alert Pipeline plugin ingests alerts from external monitoring systems and network devices through multiple protocols, normalizes them into a common format, applies deduplication to reduce noise, and routes them through configurable pipeline rules that determine what action to take -- notify a team, create an incident, trigger automation, or suppress.

Ingestion Sources

• Webhooks -- receive HTTP POST payloads from monitoring tools (Prometheus Alertmanager, Grafana, Datadog, PagerDuty, custom systems)
• Syslog -- receive syslog messages (RFC 5424/3164) from network devices and servers via UDP or TCP
• SNMP traps -- receive SNMPv2c and SNMPv3 trap notifications from network devices

How It Works

Alert Processing Pipeline

1. Ingestion -- alerts arrive via webhook, syslog, or SNMP trap endpoints. Each source has a dedicated receiver that parses the incoming data.
2. Normalization -- raw alerts are transformed into a common schema with fields for severity, source, message, device, timestamp, and custom labels.
3. Deduplication -- duplicate alerts within a configurable time window are merged. Deduplication matches on source, device, and alert signature to prevent alert storms.
4. Pipeline rules -- normalized alerts are evaluated against ordered pipeline rules. Each rule has match conditions (severity, source, device pattern, label selectors) and actions (notify, create incident, run automation, suppress, escalate).
5. Alert lifecycle -- alerts transition through states: Open → Acknowledged → Resolved. Acknowledgment and resolution can be manual or automatic.

Deduplication Strategy

Deduplication uses a configurable time window (default 5 minutes) and field matching. When a duplicate alert arrives within the window, the existing alert's count is incremented and its last-seen timestamp is updated instead of creating a new alert.

Pipeline Rule Evaluation

Rules are evaluated in order (top to bottom). The first matching rule determines the action. If no rule matches, the default action is applied (configurable: notify all, suppress, or create incident).

Step-by-Step Guide

Workflow 1: Enable the Alert Pipeline Plugin

1. Navigate to Administration → Plugins
2. Find Alert Pipeline and click Enable
3. Configure the webhook port (default 8443) and optionally enable syslog and SNMP trap receivers
4. Click Save & Start

Workflow 2: Configure a Webhook Ingestion Endpoint

1. In the Alert Pipeline settings, navigate to Ingestion Sources
2. Click Add Webhook Source
3. Name the source (e.g., "Prometheus Alertmanager") and generate an API key
4. Copy the webhook URL and API key to your monitoring tool's notification configuration

Workflow 3: Create a Pipeline Rule

1. Navigate to Alert Pipeline → Rules
2. Click Add Rule
3. Define match conditions: severity (critical, warning, info), source pattern, device pattern, label selectors
4. Set the action: notify team, create incident, run automation task, suppress, or escalate
5. Set the rule priority (order in the evaluation chain)
6. Click Save and test with a sample alert

Workflow 4: Test the Pipeline

1. Navigate to Alert Pipeline → Test
2. Paste a sample alert payload or use the built-in test generator
3. Click Send Test Alert -- the alert flows through the pipeline and the matched rule is highlighted
4. Verify the alert appears in the Alert Dashboard with the correct severity and routing

Code Examples

Webhook Payload Format

# Send an alert via webhook
curl -X POST https://netstacks.example.com:8443/api/v1/alerts/webhook \
  -H "Content-Type: application/json" \
  -H "X-API-Key: alert_key_abc123" \
  -d '{
    "source": "prometheus",
    "severity": "critical",
    "device": "core-rtr-01.dc1",
    "message": "BGP peer 10.0.0.2 down for 5 minutes",
    "labels": {
      "protocol": "bgp",
      "neighbor": "10.0.0.2",
      "asn": "65001"
    },
    "timestamp": "2025-12-15T14:30:00Z"
  }'

Syslog Configuration

# rsyslog forwarding configuration
# Forward network device syslog to NetStacks Alert Pipeline
*.* @netstacks.example.com:1514;RSYSLOG_SyslogProtocol23Format

# Cisco IOS syslog configuration
logging host 10.0.100.50 transport udp port 1514
logging trap warnings
logging source-interface Loopback0

SNMP Trap Forwarding

# Cisco IOS SNMP trap configuration
snmp-server enable traps bgp
snmp-server enable traps config
snmp-server enable traps entity
snmp-server host 10.0.100.50 version 3 priv netstacks-traps udp-port 1162

Pipeline Rule Definition

{
  "name": "Critical BGP alerts to NOC",
  "priority": 1,
  "match": {
    "severity": "critical",
    "labels": {
      "protocol": "bgp"
    },
    "device_pattern": "core-*"
  },
  "actions": [
    {
      "type": "notify",
      "target": "team:noc",
      "channels": ["slack", "email"]
    },
    {
      "type": "create_incident",
      "priority": "P1",
      "assignee": "network-ops"
    }
  ],
  "deduplication": {
    "window_seconds": 300,
    "match_fields": ["device", "labels.neighbor"]
  }
}

Test Alert via CLI

# Send a test alert to verify pipeline configuration
curl -X POST https://netstacks.example.com:8443/api/v1/alerts/test \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_token" \
  -d '{
    "severity": "warning",
    "device": "dist-sw-01.dc1",
    "message": "Interface Ethernet1/24 flapping",
    "labels": { "interface": "Ethernet1/24", "protocol": "link" }
  }'

# Response shows which rule matched
# {
#   "matched_rule": "Link flap alerts to network team",
#   "actions_taken": ["notify:team:network-ops"],
#   "alert_id": "alert_test_001"
# }

Terminal Integration

Starting with v0.0.9, alerts can be viewed and triaged directly from the NetStacks Terminal without opening the Admin UI. The Terminal activity bar includes an Alerts panel with full access to your alert data.

AlertDetailTab

Click any alert in the sidebar list to open it in a detail tab with the following capabilities:

• Triage Verdicts — assign a triage verdict (actionable, noise, duplicate, known issue) directly from the detail view to quickly categorize incoming alerts
• Timeline — view the full alert lifecycle timeline including state transitions, acknowledgments, and related events
• Actions — acknowledge, resolve, or escalate alerts with one click. Trigger associated automation tasks or create incidents from the alert context.

State Filtering

The Alerts sidebar panel supports filtering by alert state (Open, Acknowledged, Resolved) and severity level. Use the state filter to focus on active alerts that need attention. The panel polls for new alerts automatically so you see updates in real time.

Q&A

Q: What alert sources are supported?

A: The Alert Pipeline supports three ingestion methods: HTTP webhooks (compatible with Prometheus Alertmanager, Grafana, Datadog, and custom sources), syslog (RFC 5424 and RFC 3164 via UDP/TCP), and SNMP traps (v2c and v3). Each source is configured independently with its own credentials and parsing rules.

Q: How does deduplication work?

A: Deduplication merges duplicate alerts within a configurable time window (default 5 minutes). Duplicates are identified by matching the source, device, and configurable alert fields (e.g., neighbor IP, interface name). When a duplicate arrives, the existing alert's count is incremented and last-seen timestamp is updated.

Q: How do I create a pipeline rule?

A: Navigate to Alert Pipeline → Rules and click Add Rule. Define match conditions (severity, source, device pattern, label selectors) and actions (notify, create incident, run automation, suppress). Rules are evaluated in priority order -- the first matching rule determines the action.

Q: Can I route alerts to different teams?

A: Yes. Pipeline rules support team-based routing. Set the notify action target to a specific team (e.g., team:noc or team:network-ops) and select notification channels (Slack, email, webhook). Different rules can route to different teams based on severity, device, or labels.

Q: How do I test the alert pipeline?

A: Use the built-in test tool at Alert Pipeline → Test. Paste a sample alert payload and click Send Test Alert. The tool shows which pipeline rule matched and what actions were taken. You can also send test alerts via the API using the test endpoint.

Q: What is the alert lifecycle?

A: Alerts transition through three states: Open (newly created), Acknowledged (someone is working on it), and Resolved (issue is fixed). Transitions can be manual (user clicks Acknowledge/Resolve) or automatic (auto-resolve after a configurable timeout, or when a recovery alert is received).

Troubleshooting

Alerts Not Arriving

• Verify the Alert Pipeline plugin is Active in Administration → Plugins
• Check that the webhook port (default 8443) is reachable from the sending system -- test with curl
• For syslog, verify UDP/TCP traffic on the configured port is allowed through firewalls
• Check the API key or authentication token is correct for webhook sources

Deduplication Too Aggressive or Too Loose

• Adjust the deduplication time window -- shorter windows create more alerts, longer windows merge more
• Add or remove match fields to control what constitutes a "duplicate" (e.g., matching on interface name vs. device only)

Pipeline Rules Not Matching

• Use the Test tool to send a sample alert and see which rule matched (or why none matched)
• Check rule priority order -- a higher-priority rule may be matching first
• Verify label selectors match the exact label keys and values in the incoming alert

Webhook Endpoint Unreachable

• Verify the webhook port is not blocked by a firewall or in use by another service
• Check TLS certificate validity if using HTTPS
• Review plugin logs for connection errors at Administration → Audit Logs

Related Features

• Plugin System -- understand the plugin architecture that powers Alert Pipeline
• Incidents & ITSM -- automatically create incidents from critical alerts
• Scheduled Tasks -- trigger automation tasks from pipeline rule actions
• Task Monitoring -- monitor alert-triggered automation execution