Lessons (6, study in order)

① SNMP served for twenty years -- why replace it

Beginner · about 9 min

Goal: Understand that moving from polling to streaming is not "a different protocol" but a way around three mathematically unavoidable limits.

What polling does

The manager asks the device every T seconds: "what are these few hundred OIDs right now?"

★ It sounds fine, and it genuinely served for two decades -- because links were 100 Mbps and 5-minute granularity was enough.

★★ But after speeds grew three orders of magnitude, this model starts breaking in three places at once.

★★★ Limit 1: the interval decides what you can see

A 3-second 10Gbps burst polled every 30 seconds is averaged down to 1Gbps -- a 10x understatement.

★★★ The danger is that there is a bump on the chart, so you never doubt the data and instead reach the completely wrong conclusion "peak 1Gbps, plenty of headroom".

★ This is not "insufficient precision" -- the conclusion is inverted.

★★★ Limit 2: a 32-bit counter wraps every 3.44 seconds

32 bits holds 4.29GB, and 10Gbps is 1.25GB per second -- 4.29 / 1.25 ≈ 3.44 seconds.

★★ Polling every 30 seconds means it wrapped over 8 times in between, so the delta does not represent traffic at all.

★ And this limit cannot be fixed by tuning the interval: avoiding a wrap on a 10G port means going under 3 seconds -- already beyond what polling can sustain.

★★ Limit 3: seeing more detail crushes the device

Polling cost is inversely proportional to the interval: halve it and the load doubles. And an SNMP agent competes for the same CPU as the routing protocols.

★★★ So "more detail" and "don't crash the device" are directly opposed -- every device has a hard floor on how tight the interval can go.

★ On the load page: this device's SNMP floor is 4 seconds, while capturing that 3-second burst needs 3. Close, but out of reach.

How streaming solves all three at once

★★★ The key is not "faster" but that it moves the decision of when to sample from the manager to the device:

· no request-response round trip -> intervals can go sub-second (limits 1 and 3)
· the device stamps the time -> computing a rate needs no guess (jitter disappears)
· state can be "pushed on change" -> zero steady-state cost plus millisecond notification (limit 3)

★ So this is a different approach, not the same thing done faster.

Key takeaways

• ★★★ Polling has three unavoidable limits: the interval bounds visibility, 32-bit counters wrap, and cost is inversely proportional to the interval.
• ★★ What streaming really changes is who decides the sampling instant -- hand it to the device and all three loosen at once.
• ★ Two of the three cannot be fixed by a smaller interval, so "more detail" sometimes means changing mechanism.

② ★★★ One formula: observed peak = true peak x (duration / interval)

Beginner · about 10 min

Goal: Work out aliasing properly, and separate "a counter averaged down" from "a state event missed" -- two completely different distortions.

Start with the formula

For a counter: observed peak = true peak x (event duration / interval), capped at 1x.

★ For a 3-second 10Gbps burst:
· 1s interval -> you see 10 Gbps (complete)
· 10s interval -> you see 3 Gbps (3.3x understated)
· 30s interval -> you see 1 Gbps (10x understated)
· 300s interval -> you see 0.1 Gbps (100x understated)

★★★ Why "visible but false" beats "invisible" for danger

If you cannot see it, you investigate and you doubt the data.

★★★ But a plausible wrong number you do not doubt. "Peak 1Gbps, 10G link, plenty of room" is self-consistent, credible, and completely wrong.

★ The real consequence: the switch's drop counter climbs while the traffic chart shows plenty of headroom, and the whole investigation heads the wrong way.

★★ State and gauges follow different logic

★★★ A counter accumulates (the bytes are still there, just spread out), whereas state and gauges do not -- a sample happens at an instant and either lands inside or misses entirely.

★ So state has no "average" middle ground: a 2-second flap under 60-second polling is caught with probability 2/60 ≈ 3.3%, meaning a 97% chance of no trace whatsoever.

★★ Conflating the two (computing an "average" for a flap, say) produces absurd conclusions.

Microbursts: invisible even at second granularity

★ Switch buffer overruns typically happen on millisecond microbursts. By the formula above, a 1ms burst is understated 1000x even at a 1-second interval.

★★ So microbursts are invisible to any sampling-based approach -- catching them requires the device's buffer high-water marks or drop counters, not a traffic chart.

Key takeaways

• ★★★ For counters: observed peak = true peak x (duration/interval). A 3s burst at 30s sampling is understated 10x.
• ★★★ State and gauges do not accumulate -- there is no "average" middle ground, only caught or completely missed.
• ★★ "Visible but false" is more dangerous than invisible, because nobody doubts a plausible wrong number.

③ ★★★ A 32-bit counter wraps every 3.44 seconds on 10G

Advanced · about 8 min

Goal: Compute the wrap time, and understand why this limit cannot be fixed by tuning the interval.

The arithmetic

A 32-bit counter holds 2^32 bytes = 4.29 GB.

10Gbps is 10e9/8 = 1.25 GB per second.

4.29 / 1.25 ≈ 3.44 seconds -- one wrap every 3.44 seconds at line rate.

At other speeds

· 100 Mbps -> 5.7 minutes (tolerable)
· 1 Gbps -> 34 seconds
· 10 Gbps -> 3.44 seconds
· 100 Gbps -> 0.34 seconds
· 400 Gbps -> 0.09 seconds

★★★ So a 32-bit counter on a 400G port wraps more than ten times a second -- no sampling can read a meaningful delta from it.

★★ What a wrap looks like in the data

It appears as "the counter suddenly got smaller".

★ Tools handle that differently:
· some read it as a device reboot (and discard the point)
· some produce a negative rate
· some add 2^32 to compensate -- but how many times it wrapped is unknowable, so that only works when it happened to wrap exactly once

★★★ The key conclusion: once the interval exceeds the wrap time, no post-processing can recover the data.

There is only one direction for a fix

★ 64-bit counters (ifHC* in SNMP; OpenConfig is 64-bit throughout):
· 468 years to wrap on a 10G port
· still 11.7 years on 400G

★★★ Note this limit cannot be dodged by tuning the interval: avoiding a wrap on 10G means going under 3 seconds, beyond what polling sustains (this device's floor is 4 seconds). So the answer is change the counter, not the setting.

Key takeaways

• ★★★ 4.29GB of 32-bit range / 1.25GB per second at 10Gbps = a 3.44-second wrap. Just 0.34s on 100G.
• ★★ Wrapped data cannot be recovered in post-processing, because the wrap count itself is lost.
• ★ The fix is 64-bit counters (468 years on 10G), not a smaller sampling interval.

④ ★★★ ON_CHANGE for state, SAMPLE for counters

Advanced · about 9 min

Goal: Master the subscription-mode rule, and understand why getting it backwards has asymmetric consequences.

First classify the data

· counters (bytes, packets) -> monotonically increasing, change with every packet
· state (interface up/down, BGP neighbour state) -> normally does not change, a few times a day
· gauges (CPU, temperature, optical power) -> a current value moving up and down

★ This is not an academic taxonomy -- it directly decides the subscription mode.

★★★ Why ON_CHANGE is the perfect answer for state

"Push only when the value actually changes" delivers two things that normally conflict:

· cheap: near-zero traffic in steady state (nothing changed, nothing sent)
· fast: millisecond notification on change, with no period to wait for

★★★ Polling has to trade one against the other: fast means dense, and dense means expensive. ON_CHANGE steps outside the trade entirely.

★ For that 2-second flap: polling misses it 97% of the time, while ON_CHANGE certainly records both transitions.

★★★ But getting it backwards is asymmetric

· state as SAMPLE -> tens of thousands of pointless questions (same answer every time) plus a period of delay. Wasteful and slow, but harmless.

· a counter as ON_CHANGE -> the counter changes with every packet, so this demands a push per packet. ★★★ That crushes both device and collector -- a config error that directly causes outages.

★ So the first step in subscription design is always deciding whether a leaf is a counter or a state.

Production uses mixed subscriptions

★ A single device normally runs several subscriptions at once:
· state leaves -> ON_CHANGE
· counter leaves -> SAMPLE, with the interval set by how short a burst you need to catch
· gauges -> SAMPLE, with the interval matched to how fast they move (minutes for temperature, seconds for CPU)

★★ TARGET_DEFINED hands that judgement to the device -- convenient, but vendors may decide differently, so one config behaves differently on two platforms and is awkward to troubleshoot.

Key takeaways

• ★★★ The rule: ON_CHANGE for state (cheap and fast), SAMPLE for counters (they change every packet).
• ★★★ The mistakes are asymmetric: state as SAMPLE only wastes; a counter as ON_CHANGE kills the device.
• ★ Production runs mixed subscriptions -- several modes on one device, not an either/or.

⑤ ★★★ When the monitoring causes the outage

Troubleshooting · about 9 min

Goal: Understand the cost of believing "polling is read-only so it is harmless", and the correct order of response.

The incident script

★★★ This script recurs in production:

(1) something inexplicable happens and "our monitoring is too coarse"
(2) the poll interval goes from 5 minutes to 30 seconds (feels like just "10x finer")
(3) the device's control-plane CPU pins at 100%
(4) BGP neighbours drop, BFD false-positives, routing reconverges network-wide
(5) and the reconvergence burns more control plane -> worse -> positive feedback

Why polling is so expensive

For one GetBulk round trip the device must: parse the PDU -> walk the MIB tree by OID -> fetch each value -> encode it back.

★★★ All of it on the control-plane CPU, competing with route computation for the same resource.

★ So "polling is a read-only operation, therefore harmless" is a dangerous misconception -- it is real load, and it competes with the most critical processes you have.

★★ Don't forget the collector side

Going from a 5-minute interval to 10 seconds multiplies data points by 30.

★ The device may cope; the collector and time-series database may not. And an overloaded collector looks a lot like "the device isn't sending" (gaps in the data), so monitor the collector's drop counters and write latency separately.

★★ Same thinking as the "three counters" method in the xFlow module: the sender's counters are not enough; you need how much the receiver successfully processed.

The correct order of response

★★★ When you need finer granularity, the order should be:

(1) ask which events you need to see (how short a burst? how fast a flap?)
(2) compute the interval that requires (within the event's duration)
(3) check whether the current mechanism sustains that interval
(4) if not, change mechanism (gNMI) rather than pushing the same one to its limit

★ Doing it backwards (tune first, observe later) is the opening of that incident script.

Key takeaways

• ★★★ An SNMP agent competes for the same CPU as the routing protocols -- polling is not "harmless because read-only".
• ★★ The collector side overloads too, and it looks like "the device isn't sending".
• ★ Correct order: name the events -> compute the interval -> check sustainability -> change mechanism if needed.

⑥ ★★★ A dead stream looks like "all quiet"

Troubleshooting · about 9 min

Goal: Recognise the new failure mode streaming telemetry introduces, and why it is more dangerous than a failed poll.

Two completely different failure shapes

★★★ · a failed poll -> the request times out, the manager knows immediately, and usually alarms
· a stopped stream -> nothing arrives, and "nothing arriving" is indistinguishable in the data from "nothing changed"

★ So: the device is up, nothing alarms, and the chart is a flat line -- identical to "the network is quiet".

★★ Why it drops

gNMI runs over a long-lived connection (gRPC over HTTP/2). Plenty of reasons it breaks:

· the collector restarts or is upgraded
· a stateful middlebox (firewall, NAT) ages out the idle connection
· the device's subscription process restarts without re-establishing
· the TLS certificate expires

★ Note the last two: the device may itself "believe it is still pushing", so even show subscription on the device may not reveal it.

★★★ So you must monitor the subscription itself

★★★ Adopting streaming telemetry requires one extra monitor: is the subscription alive.

The practical approach is a heartbeat test: "has any data point at all arrived from this device in the last N seconds" -- alarm if not, regardless of whether "the value didn't change".

★ This is why many platforms attach one low-frequency SAMPLE leaf to each subscription as a heartbeat: even with everything else on ON_CHANGE, something must arrive regularly to prove the pipe is open.

A more general lesson

★★★ This outage is the same family as the missing IPFIX template in the xFlow module:

"I sent it" is not "they received and understood it".

★ So monitoring any data path by the sender's counters alone is never enough -- you need how much the receiver successfully processed.

★★ And for a monitoring system there is one more layer: the monitoring needs monitoring too, and it cannot be monitored by itself.

Key takeaways

• ★★★ A stopped stream is silent: flat chart, no alarm, identical to "the network is quiet".
• ★★ So build a heartbeat monitor -- "any data point in the last N seconds" -- rather than waiting for an alarm.
• ★ More generally: "I sent it" is not "they received and understood it", and the monitoring system itself needs monitoring.