Stacks & Deployments

Enterprise

Bundle config templates into stacks, render them with Jinja2, and roll them out with a backed-up, atomic deployment pipeline tracked through real status states.

Overview

Stacks & Deployments is a core NetStacks Controller configuration feature (it is not a plugin). It lets you bundle multiple config templates into a reusable stack, capture concrete variable values as a stack instance, and push the rendered configuration to target devices through a backed-up, optionally atomic deployment pipeline. All endpoints live under the Controller's /api/config namespace alongside the rest of the configuration management API.

A stack is an ordered list of services, where each service references one config template by ID and supplies the variables that template needs. For example, a "Branch Office" stack might chain a router template, a switch template, and a firewall template into a single deployable unit, sharing site-wide variables such as the site name and management subnet.

Enterprise / Controller feature

Stacks & Deployments requires the NetStacks Controller. It is backed by the Controller's configuration tables (config_stacks, config_stack_instances, and config_deployments) and the shared config template store, not by any plugin migrations.

Key Concepts

Concept	Description
Config Template	A single Jinja2 source with a platform, config format (CLI/JSON), target path, and operation (replace/merge). Managed under `/api/config/templates`.
Stack	An ordered set of `services`, each referencing a template by `template_id` with its own variables. Has an `atomic` flag and optional `deployment_procedure` for MOP-gated rollouts.
Stack Instance	A saved, undeployed selection of target devices plus `variable_values` and per-device `device_overrides`. Lives until you deploy it.
Deployment	An execution record that renders the stack, backs up devices, pushes configs, and tracks per-device status. Created from an instance via `/deploy`.

Data Model

Understanding the three records helps you map the UI and API to what is actually stored.

Stack

A stack has a name, optional description, an atomic boolean (defaults to true), and a services array. Each service entry references a template and binds variables:

stack.jsonjson

{
  "name": "Branch Office",
  "description": "Router + switch for a standard branch",
  "atomic": true,
  "services": [
    {
      "template_id": "0b6e...router-template-uuid",
      "order": 0,
      "variables": { "site_name": "{{ site_name }}", "wan_ip": "{{ wan_ip }}" }
    },
    {
      "template_id": "9c2a...switch-template-uuid",
      "order": 1,
      "variables": { "site_name": "{{ site_name }}", "data_vlan": 100 }
    }
  ]
}

Stack Instance

An instance binds a stack to real devices and concrete variable values. variable_values are free-form JSON (untyped key / value pairs), and device_overrides hold per-device values that win over the shared values at render time.

instance.jsonjson

{
  "stack_id": "stack-uuid",
  "name": "NYC-Branch-01",
  "target_devices": { "0": ["rtr-device-uuid"], "1": ["sw-device-uuid"] },
  "variable_values": {
    "site_name": "NYC-BR01",
    "wan_ip": "203.0.113.2",
    "data_vlan": 100
  },
  "device_overrides": {
    "rtr-device-uuid": { "wan_ip": "203.0.113.6" }
  }
}

Variables are untyped JSON

There is no typed/validated variable schema (no enforced ip_address/cidr/select types). Variables are JSON values rendered through Jinja2. Add your own validation inside templates using Jinja2 tests and filters if you need it.

Deployment

A deployment record carries a status, target_devices, the variable_values and device_overrides used, and rollup counters (total_devices, succeeded_count, failed_count). Per-device rows track each push individually, including the backup_config_idcaptured before the change.

Deployment Pipeline

When you deploy an instance, the Controller runs a six-stage pipeline as a background task and streams progress into the deployment logs.

Render — resolve the stack's services into per-device render jobs, applying shared variables and per-device overrides through Jinja2.
Pre-flight backup — pull the current config from each target device and store it. Both a structured backup (via gNMI/NETCONF) and a CLI backup (via SSH) are captured, giving a complete restore point before any change is applied.
Push — apply rendered configs. Devices are pushed in parallel (bounded concurrency), serial within a device.
Validate — compare the pushed config against the device's actual state to detect drift.
Confirm — finalize confirmed commits, or let them expire so the device auto-rolls-back on drift.
Sync — pull the full post-deploy config so history reflects the device's new state.

Deployment Status Values

A deployment moves through these real status values (stored on the config_deployments record):

Status	Meaning
`pending`	Record created; execution not yet started.
`in_progress`	The pipeline is running (rendering, backing up, pushing).
`completed`	All target devices succeeded.
`failed`	One or more devices failed to apply.
`rolled_back`	The deployment was reverted to the pre-flight backup (via the rollback endpoint, or automatically on an atomic stack).

System-created deployments generated from a MOP procedure may start in an approved state before execution. There is no built-in "pending approval" step inside this status machine itself — approval gating is handled by the separate MOP / Approvals feature (see below).

Atomic Rollback

All-or-nothing deploys

When a stack has atomic: true (the default) and any device fails during the push, the Controller automatically rolls the already-succeeded devices back to their pre-flight backup. The deployment then ends in rolled_back so the network is left in a consistent state rather than a partial one.

Step-by-Step Guide

Workflow 1: Create a Stack

First create the individual config templates you want to chain (each with a Jinja2 source, platform, and config format).
Create a stack and add one service per template, in deployment order, binding the variables each template expects.
Decide whether the stack should be atomic (recommended) so a partial failure rolls back automatically.

Workflow 2: Create and Preview an Instance

Create an instance from the stack, selecting target devices.
Supply variable_values for the site, plus any device_overrides for individual devices.
Call the stack render preview to dry-run the Jinja2 output for every device without deploying. Verify substitution and logic before pushing.

Workflow 3: Deploy

Deploy the instance directly with the deploy endpoint, or deploy via a MOP procedure for change-controlled, approval-gated rollouts.
Watch progress in the deployment logs as it moves through render, backup, push, validate, confirm, and sync.
On success the deployment reaches completed; on failure an atomic stack reaches rolled_back.

Workflow 4: Inspect or Roll Back

Fetch the deployment detail to see per-device status and counters.
Read the deployment logs for stage-by-stage messages.
Trigger the rollback endpoint to restore the pre-flight backups captured at deploy time.

Tip

Always run the render preview before deploying. It renders every device's config through Jinja2 with the instance's variables and overrides, so you catch undefined variables or logic errors before they reach a device.

Variables & Overrides

Templating is Jinja2 throughout (the Controller uses a MiniJinja engine with a few Python-friendly helpers). Variables are resolved in two scopes by the stack resolver:

Shared values — instance-level variable_values applied to every device.
Per-device overrides — device_overrides keyed by device ID; these win over shared values for that device only.

API Variables (dynamic lookups)

A stack's variable_config can map a variable to an external API resource (a resource_id, HTTP method, path, and a json_path to extract). The resolve-variables endpoint fetches these at prepare time and returns shared and per-device values you can fold into the deployment. This replaces the older idea of inline mustache device placeholders — there is no separate mustache layer; everything renders through Jinja2.

variable_config.jsonjson

{
  "wan_ip": {
    "resource_id": "ipam-resource-uuid",
    "method": "GET",
    "path": "/addresses/?device={{ device.hostname }}",
    "json_path": "results.0.address",
    "scope": "per_device"
  }
}

Using device context in templates

Inside a Jinja2 template you can reference values that the resolver puts in scope, including device attributes. Use standard Jinja2 access:

router.j2jinja2

hostname {{ device.hostname }}
!
{% if data_vlan is defined %}
vlan {{ data_vlan }}
 name DATA
{% endif %}
!
interface GigabitEthernet0/0
 description Uplink for {{ site_name }}
 ip address {{ wan_ip }} {{ wan_mask | default("255.255.255.252") }}

API Examples

All routes are mounted under the Controller's /api/config namespace. Individual config templates live at /api/config/templates; stacks, instances, and deployments live at the paths below.

Create a Stack

create-stack.shbash

curl -X POST https://controller.example.com/api/config/stacks \
  -H "Authorization: Bearer your_token" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "Branch Office",
  "description": "Router + switch for a standard branch",
  "atomic": true,
  "services": [
    {
      "template_id": "ROUTER_TEMPLATE_UUID",
      "order": 0,
      "variables": { "site_name": "{{ site_name }}", "wan_ip": "{{ wan_ip }}" }
    },
    {
      "template_id": "SWITCH_TEMPLATE_UUID",
      "order": 1,
      "variables": { "site_name": "{{ site_name }}", "data_vlan": 100 }
    }
  ]
}'

Preview the Rendered Stack (dry-run)

render-preview.shbash

curl -X POST https://controller.example.com/api/config/stacks/STACK_ID/render \
  -H "Authorization: Bearer your_token" \
  -H "Content-Type: application/json" \
  -d '{
  "target_devices": { "0": ["RTR_DEVICE_ID"], "1": ["SW_DEVICE_ID"] },
  "variable_values": { "site_name": "NYC-BR01", "wan_ip": "203.0.113.2" },
  "device_overrides": {}
}'

# Response: render jobs (no deploy), one per device/service
{
  "jobs": [
    {
      "device_id": "RTR_DEVICE_ID",
      "device_name": "nyc-br01-rtr01",
      "template_name": "Branch Router",
      "service_order": 0,
      "rendered_config": "hostname nyc-br01-rtr01\n...",
      "target_path": "/",
      "operation": "replace",
      "config_format": "cli"
    }
  ]
}

Resolve API Variables

resolve-variables.shbash

curl -X POST https://controller.example.com/api/config/stacks/STACK_ID/resolve-variables \
  -H "Authorization: Bearer your_token" \
  -H "Content-Type: application/json" \
  -d '{ "target_devices": ["RTR_DEVICE_ID", "SW_DEVICE_ID"] }'

Create an Instance

create-instance.shbash

curl -X POST https://controller.example.com/api/config/instances \
  -H "Authorization: Bearer your_token" \
  -H "Content-Type: application/json" \
  -d '{
  "stack_id": "STACK_ID",
  "name": "NYC-Branch-01",
  "target_devices": { "0": ["RTR_DEVICE_ID"], "1": ["SW_DEVICE_ID"] },
  "variable_values": {
    "site_name": "NYC-BR01",
    "wan_ip": "203.0.113.2",
    "data_vlan": 100
  },
  "device_overrides": {
    "RTR_DEVICE_ID": { "wan_ip": "203.0.113.6" }
  }
}'

Deploy an Instance

deploy-instance.shbash

curl -X POST https://controller.example.com/api/config/instances/INSTANCE_ID/deploy \
  -H "Authorization: Bearer your_token"

# Response: a deployment record (execution runs in the background)
{
  "id": "DEPLOYMENT_ID",
  "stack_id": "STACK_ID",
  "name": "NYC-Branch-01",
  "status": "pending",
  "total_devices": 2,
  "succeeded_count": 0,
  "failed_count": 0
}

Deploy via a MOP (change-controlled)

deploy-with-mop.shbash

curl -X POST https://controller.example.com/api/config/instances/INSTANCE_ID/deploy-with-mop \
  -H "Authorization: Bearer your_token" \
  -H "Content-Type: application/json" \
  -d '{ "control_mode": "step", "name": "NYC-BR01 cutover" }'

Track, Log, and Roll Back

track-and-rollback.shbash

# List deployments (optionally filtered by stack)
curl https://controller.example.com/api/config/deployments \
  -H "Authorization: Bearer your_token"

# Deployment detail with per-device status
curl https://controller.example.com/api/config/deployments/DEPLOYMENT_ID \
  -H "Authorization: Bearer your_token"

# Stage-by-stage logs
curl https://controller.example.com/api/config/deployments/DEPLOYMENT_ID/logs \
  -H "Authorization: Bearer your_token"

# Restore pre-flight backups
curl -X POST https://controller.example.com/api/config/deployments/DEPLOYMENT_ID/rollback \
  -H "Authorization: Bearer your_token"

Q&A

Q: Is Stacks & Deployments a plugin?: A: No. It is a core NetStacks Controller configuration feature, served under /api/config and backed by the Controller's config_stacks, config_stack_instances, and config_deployments tables. It does not depend on the plugin system.

Q: How is a stack different from a template?: A: A config template is a single Jinja2 source with one platform, format, target path, and operation. A stack is an ordered set of services, each pointing at a template by template_id and supplying its variables — so a stack chains a router, switch, and firewall template into one deployable unit with shared variables.

Q: What deployment statuses are there?: A: pending → in_progress → completed, or failed, or rolled_back. MOP-generated deployments may begin in an approved state. There is no separate "deploying" or "deployed" status — use in_progress and completed.

Q: Are configs backed up before a change?: A: Yes. The pre-flight stage pulls each device's current config (structured via gNMI/NETCONF and CLI via SSH) and stores it before anything is pushed. The backup is linked to the device deployment so rollback can restore exactly that state.

Q: What does "atomic" mean?: A: With atomic: true (the default), if any device fails during the push the Controller automatically rolls the already-succeeded devices back to their pre-flight backups and ends the deployment in rolled_back, so you never end up with a half-applied stack.

Q: Do deployments require approval?: A: Approval is not part of the deployment status machine itself. Use the MOP path (/deploy-with-mop against a stack with a deployment procedure) when you need change control and approval gating. See the MOP Approvals and Method of Procedures docs.

Q: Are variables typed and validated?: A: No. variable_values and device_overrides are free-form JSON rendered through Jinja2. There is no enforced ip_address/cidr/select type system — add validation with Jinja2 tests/filters inside your templates if you need it.

Q: Can I preview before deploying?: A: Yes. POST /api/config/stacks/:id/render returns the fully rendered config for every device/service as a dry-run, without touching any device.

Troubleshooting

Template Rendering Errors

Run POST /api/config/stacks/:id/render first; the response surfaces resolution and Jinja2 errors before deploy.
Common Jinja2 issues: undefined variables, unbalanced {% ... %} blocks, and incorrect filter usage. Guard optionals with {% if var is defined %} or the default() filter.
Ensure every variable a service references is provided in the instance's variable_values or a matching device_overrides entry.

Deployment Stuck in in_progress

Read GET /api/config/deployments/:id/logs for the stage that stalled (render, backup, push, validate, confirm, sync).
Verify reachability and credentials for the target devices — SSH/gNMI/NETCONF access must work from the Controller, with write/config privileges.
A failed pre-flight backup stops the deploy by design; fix device connectivity so a restore point can be captured first.

Partial Failures

On an atomic stack, a single device failure rolls back the rest and the deployment ends rolled_back. Inspect per-device rows in the deployment detail for the specific error_message.
For non-atomic stacks, succeeded devices keep their config and the deployment is marked failed; use the rollback endpoint if you need to revert them.

Rollback

POST /api/config/deployments/:id/rollback restores the pre-flight backups captured at deploy time.

Warning

Rollback re-pushes the pre-flight configuration to the affected devices. Confirm that snapshot is still valid for the current network state before initiating it.

Creating Stacks — build a stack from config templates and tune the atomic flag.
Stack Instances — select devices and capture variable values before deploying.
Variable Overrides — shared vs. per-device variable scopes.
Deployment History — review past deployments, status, and rollbacks.
Jinja2 Syntax — the templating language used to render stack configs.
Method of Procedures — change-controlled, MOP-gated deployments.
MOP Approvals — approval gating for deployments that need sign-off.
Stacks API — full API reference for stacks and deployments.