Why NSO projects stall, and the 7 fixes

Most NSO projects don't fail. They stall, quietly, around month six.

Nobody cancels anything. The licenses get renewed, the server stays up, and there's still a line for it in the roadmap. But new services stop shipping, the people who were excited about it move on to other work, and changes go back to being typed into devices by hand. When we are asked to look at one of these projects, the cause is almost always some mix of the same seven patterns. Each one leaves a trace in NSO itself.

1. The first service was too big

The first service tried to cover every variant: every platform, every edge case, every option a customer has ever ordered. Months later it still isn't done, and nothing is in production to show for the work.

admin@ncs# show packages package l3vpn package-version
packages package l3vpn
 package-version 0.0.14
admin@ncs# show running-config services l3vpn | count
Count: 0 lines

Fourteen package versions and no instances. The model has been redesigned more times than it has been used.

The fix is to ship one narrow service within 30 days. Pick something frequent, low-risk, and boring, like a VLAN or an interface description. The whole model fits on a screen:

module vlan {
  namespace "http://example.net/vlan";
  prefix vlan;
  import tailf-ncs { prefix ncs; }

  list vlan {
    key name;
    uses ncs:service-data;
    ncs:servicepoint vlan-servicepoint;

    leaf name    { type string; }
    leaf vlan-id { type uint16 { range "1..4094"; } }
    leaf-list device {
      type leafref { path "/ncs:devices/ncs:device/ncs:name"; }
    }
  }
}

A small service that operators use every day teaches a team more than a large one that's always almost finished.

2. Brownfield config never got reconciled

Half the devices show out-of-sync, and nobody wants to run sync-from because nobody is sure what it will pull in or overwrite. So NSO's view of the network drifts further from the real network every week.

admin@ncs# devices check-sync
sync-result {
    device pe1-den
    result in-sync
}
sync-result {
    device pe2-den
    result out-of-sync
}

The fix is to make drift visible and boring. Look before pulling:

admin@ncs# devices device pe2-den compare-config
diff
 devices {
     device pe2-den {
         config {
             interface GigabitEthernet0/0/3 {
-                ip access-group cust-1042-in in
             }
         }
     }
 }

That minus sign means the device has something NSO doesn't. Somebody removed an access-group by hand. Decide whether that change is wanted, then sync-from to accept it or push the service again to reassert it. Run check-sync on a schedule, publish the count, and work through a reconciliation plan device by device. Then agree on a policy for out-of-band changes, so the network doesn't drift again the week after the reconciliation finishes. The full procedure is in Reconciling out-of-sync devices in NSO.

3. NSO and NED versions drifted

Upgrades keep getting postponed because nobody knows what will break. Each skipped release makes the next upgrade bigger and scarier, until the platform is several versions behind and stuck there.

admin@ncs# show ncs-state version
ncs-state version 5.7.6
admin@ncs# show packages package package-version | include cisco-iosxr
packages package cisco-iosxr-cli-7.33

NSO 5.7 reached end of support, and a NED from the same era with it. Every bug the team hits has been fixed in a version they can't reach without a jump.

The fix is a version matrix that lists NSO, every NED, and every package, plus an upgrade path tested in netsim before it touches production. NED upgrades are a per-device action, which makes them safe to roll out in batches:

admin@ncs# devices device pe2-den migrate new-ned-id cisco-iosxr-cli-7.52 dry-run
modified-path {
    path /ncs:devices/device{pe2-den}/config/cisco-ios-xr:router/bgp
    info "Schema changed"
}

After that, upgrade on a routine cadence so each one stays small.

4. No CI or testing

Packages get edited directly on the server. There's no history, no review, and no way to know whether a change breaks a service until it does, in production.

$ ls /var/opt/ncs/packages/
vlan  vlan.bak  vlan.bak2  vlan-working

The fix is to make git the source of truth and run every change through a pipeline built on netsim devices. Before anything is committed, someone reviews the dry-run diff:

admin@ncs(config)# commit dry-run outformat native
native {
    device {
        name pe2-den
        data interface GigabitEthernet0/0/1.1042
              description vlan-1042
              encapsulation dot1q 1042
             !
    }
}

If the diff surprises the reviewer, it would have surprised everyone in production too. Building that pipeline on GitLab takes a day.

5. Nobody owns the service model

The YANG models were written by developers without much input from the operators who will use them. The models expose the wrong inputs, miss the options people actually order, and nobody has the authority to fix them.

The trace is in the data. Count the leaves operators always set to the same value, or never set at all:

admin@ncs# show running-config services l3vpn | include qos-policy | count
Count: 0 lines

A leaf that is never set is a leaf that should not be in the model.

The fix is a named owner for each service model and a model review with operators before anything gets built. Twenty minutes walking an engineer through the inputs saves weeks of rework later.

6. The one developer left

One person built most of the packages, and that person moved on. What's left works, mostly, but nobody understands it well enough to change it.

$ git shortlog -sn packages/
   214  jsmith
     3  kpatel

The fix is to document what exists, refactor the critical paths first, and pair with the team's own staff while doing it. Start with show running-config services | display service-meta-data, which shows which service owns every line of device config, because that is the map nobody drew. The work isn't finished until there's a handoff runbook the team has actually used.

7. Nothing consumes it

NSO works, but requests still arrive as tickets, and an engineer still reads each ticket and types the change. NSO is just one more tool they could use if they had time.

$ grep -c restconf /var/log/ncs/audit.log
0

No RESTCONF calls means no system is driving NSO. Only people are.

The fix is to connect NSO to the request path, through a portal, an API, or the ticketing system itself. One call per change:

$ curl -s -u "$NSO_USER:$NSO_PASS" -X POST \
    -H "Content-Type: application/yang-data+json" \
    -d '{"vlan:vlan":[{"name":"CHG0048211","vlan-id":1042,"device":["pe2-den"]}]}' \
    "https://nso.example.net/restconf/data?dry-run=native"

Run it with dry-run=native from the ticket first, attach the result to the change record, and run it for real once the change is approved. Make it the normal way changes happen in the change process, and track adoption so the team can see whether it's working.

A quick self-check

If three of these sound familiar, the project is stalled, not slow. The good news is that none of them are technical dead ends. Each one has a fix that fits into a few weeks of focused work.

If a project got stuck in a way that isn't on this list, or got moving again, tell us about it in the comments.