Reconciling out-of-sync devices in NSO without breaking anything

Every NSO estate older than a year has an out-of-sync backlog. Somebody fixed something at 2 a.m. from the console, a vendor TAC session changed a timer, a script nobody remembers pushed an ACL. NSO notices and marks the device out of sync. Then nothing happens, because the two commands that fix it both sound dangerous, and nobody is sure which one to run.

This is the procedure we use on a backlog of any size. It is slow on purpose. Speed is what caused the backlog.

What out-of-sync means

NSO keeps a copy of each device's configuration in CDB and a transaction ID, or a hash, that the device reported when it was last synchronized. check-sync asks the device for its current ID and compares:

admin@ncs# devices device pe2-den check-sync
result out-of-sync
info got: 7c1e4f9a expected: 2b90d1c3

It says the device changed. It does not say how. Anyone who runs sync-from or sync-to at this point is guessing.

By default NSO refuses to commit a change to an out-of-sync device. That is the right default, and the reason a backlog freezes automation: every service touching that device fails until someone reconciles it.

admin@ncs# show running-config devices global-settings out-of-sync-commit-behaviour
devices global-settings out-of-sync-commit-behaviour reject

Leave it on reject. Teams that flip it to accept to get unstuck are choosing not to know what the network looks like.

Step 1. Measure the backlog

Before touching anything, get the list and keep it:

admin@ncs# devices check-sync | include out-of-sync | count
Count: 61 lines
admin@ncs# show devices list | count
Count: 118 lines

Then capture every diff. compare-config shows what the device has that NSO doesn't, and what NSO has that the device doesn't, without changing either side:

$ for d in $(echo "show devices list | nomore" | ncs_cli -C -u admin | awk 'NR>1 {print $1}'); do
    echo "devices device $d compare-config" | ncs_cli -C -u admin > "diffs/$d.txt"
  done
$ wc -l diffs/*.txt | sort -n | tail -5
     4 diffs/pe1-den.txt
    12 diffs/pe3-slc.txt
    38 diffs/pe2-den.txt
   412 diffs/agg7-slc.txt

Sort by size. The small diffs are easy. The 412-line one is where the actual problem lives, and it is probably a device somebody rebuilt from a template without telling NSO.

Step 2. Read the diffs and sort them into three piles

In compare-config output, a minus sign is config present on the device and absent in NSO. A plus sign is the reverse.

admin@ncs# devices device pe2-den compare-config
diff
 devices {
     device pe2-den {
         config {
             interface GigabitEthernet0/0/3 {
-                ip access-group cust-1042-in in
             }
             ntp {
-                server 10.0.0.9
+                server 10.0.0.5
             }
         }
     }
 }

Two different things happened here. Somebody removed an access-group by hand, and somebody changed the NTP server. Each line goes in one of three piles:

1. Keep it. The out-of-band change is correct and NSO should learn it. The NTP change is probably this: the old server was decommissioned and the device was fixed by hand.
2. Reassert it. The out-of-band change is wrong, or it undid something a service owns. The missing access-group is this, if a service put it there.
3. Investigate. Nobody knows. The 412-line diff is all of this pile.

Write the pile down per device, in the ticket. This is the only part of the procedure that needs judgment, and it is the part teams try to skip.

Step 3. For "keep it": sync-from, after a dry run

sync-from pulls the device's config into CDB and overwrites NSO's copy. It never touches the device. It is safe for the network and dangerous for NSO's understanding of it, because anything NSO had that the device doesn't will disappear from CDB. Preview first:

admin@ncs# devices device pe2-den sync-from dry-run

The output is the same diff as compare-config, shown as what CDB will become. If it matches the "keep it" pile and nothing else, run it:

admin@ncs# devices device pe2-den sync-from
result true

For a large device where only one subtree changed, partial-sync-from pulls just that path and leaves the rest of CDB alone:

admin@ncs# devices partial-sync-from path [ /devices/device[name='pe2-den']/config/cisco-ios-xr:ntp ]

Step 4. For "reassert it": find the service, then re-deploy

If the removed access-group was placed by a service, NSO knows which one:

admin@ncs# show running-config devices device pe2-den config interface GigabitEthernet 0/0/3 | display service-meta-data

The output annotates each line with the service that owns it and a reference count. With the owner known, a re-deploy puts the config back. Dry run first, always:

admin@ncs# services l3vpn cust-1042 re-deploy dry-run
cli {
    local-node {
        data  devices {
                  device pe2-den {
                      config {
                          interface GigabitEthernet0/0/3 {
             +                ip access-group cust-1042-in in
                          }
                      }
                  }
              }
    }
}

Note that re-deploy on an out-of-sync device will itself be rejected. The order matters: sync-from to accept the device's current state, then re-deploy to push the service's intent back on top of it. After both, check-sync reports in-sync and the service is whole.

Step 5. For config that is already right: no-networking

Sometimes the device already has exactly what a service would produce, but NSO doesn't know the service exists. This is the brownfield case: the network predates the automation. Creating the service normally would try to push config that is already there, and the NED may or may not treat that as a no-op.

commit no-networking writes the service to CDB and marks the device config as service-owned without sending anything to the device:

admin@ncs(config)# vlan access-1042 vlan-id 1042 device pe2-den
admin@ncs(config-vlan-access-1042)# commit no-networking

Then verify the device agrees:

admin@ncs# devices device pe2-den check-sync
result in-sync

If it reports out-of-sync after a no-networking commit, the device did not actually have what the service produces, and the right move is to commit dry-run outformat native and look at the gap.

For a service that should take ownership of config that is already on the device and also already in CDB, re-deploy reconcile sets the reference counts so the service owns it and a future delete of the service removes it:

admin@ncs# services vlan access-1042 re-deploy reconcile { keep-non-service-config }

keep-non-service-config is the safe option. It means config under the service's path that the service did not produce stays on the device when the service is deleted.

Step 6. Never blanket sync-to

sync-to pushes NSO's copy to the device and overwrites whatever is there. On an out-of-sync device, that means deleting every out-of-band change at once, including the one the NOC made at 2 a.m. to keep a customer up. There is a place for sync-to: a device that was factory reset and needs its config restored. There is no place for it in a reconciliation loop.

Step 7. Make it boring

The backlog came from out-of-band changes that nothing noticed. The fix after reconciliation is to notice them within a day, not a year. A nightly job that records the count and the per-device diffs is enough:

#!/usr/bin/env bash
# /etc/cron.daily/nso-drift
set -euo pipefail
out=/var/lib/nso-drift/$(date +%F)
mkdir -p "$out"
echo "devices check-sync" | ncs_cli -C -u admin > "$out/check-sync.txt"
grep -B1 out-of-sync "$out/check-sync.txt" | awk '/device/ {print $2}' | while read -r d; do
  echo "devices device $d compare-config" | ncs_cli -C -u admin > "$out/$d.diff"
done
count=$(grep -c out-of-sync "$out/check-sync.txt" || true)
echo "$(date +%F) $count" >> /var/lib/nso-drift/history.txt

Publish the number where the team sees it. A count that goes from 61 to 0 and stays there is the proof that reconciliation worked. A count that creeps back to 10 in a month is a process problem with a name on it, and the diffs say whose.

The last piece is policy, not tooling: out-of-band changes are allowed in an emergency, the ticket must say so, and the device gets reconciled the next business day. Teams that write that down and hold to it stop having a backlog. Teams that don't will be reading this post again next year.