Lessons (6, study in order)

① Two genuinely different things: aggregation vs sampling

Beginner · about 8 min

Goal: Understand that NetFlow/IPFIX and sFlow are not "different versions" but two fundamentally different philosophies -- and why that decides which questions each can answer.

Why you need it at all

An interface counter can only tell you "this port is doing 800Mbps". It cannot answer:

· who is using those 800M?
· where is it going?
· when did it start?

★ Flow observability fills that hole: turning "how much" into "who, where, how long".

★★★ The aggregators: NetFlow / IPFIX

The device maintains an internal flow cache, aggregating packets into flows by 5-tuple: packets with the same 5-tuple keep adding to a packet and byte count, and eventually one statistics record is exported.

★ Upside: exact counts (unsampled, they are the truth) and small data volume.
★★ Price: it must wait for the flow to end (or time out) before exporting -> inherent lag; and the records are not self-describing, so the collector needs a template to read them.

★★★ The sampler: sFlow

No aggregation whatsoever. At a fixed ratio (1:1000, say) it takes a whole packet header and ships it verbatim, plus periodic interface counters.

★ Upside: device cost is tiny and constant, you see packet-level detail (TCP flags, the real packet-size distribution), and there is zero lag (sampled means shipped -- no cache).
★★ Price: every volume figure is an estimate (samples x rate).

So "which is better" is the wrong question

★★★ They answer different questions:

· need exact billing / audit -> aggregation, and unsampled
· need to see bursts within seconds -> sampling (no timeout lag)
· need packet-level detail (who is sending malformed packets) -> only sFlow can give it

★ Production commonly deploys both, each covering its own ground.

Key takeaways

• ★★★ NetFlow/IPFIX = aggregate into flows on the device; sFlow = ship sampled packet headers. Two philosophies, not two versions.
• ★★ Aggregation is exact but lags by one timeout; sampling has zero lag but every figure is an estimate.
• ★ IPFIX is the standardised NetFlow v9 -- greenfield networks should go straight to it.

② ★★★ What "1:1000 sampling" actually means

Beginner · about 10 min

Goal: Work out the real cost of a sampling rate -- it is not "one-thousandth of the precision" but an extreme filter that keeps every big flow and throws away every small one.

Start with a counter-intuitive number

The chance a flow of P packets is sampled at least once is:

P(seen) = 1 - (1 - 1/N)^P

★ Substituting N=1000:
· a 10-packet flow -> about 1%
· 100 packets -> about 10%
· 10,000 packets -> about 99.99%
· 1,000,000 packets -> effectively 100%

★★★ So 1:1000 does not mean "one-thousandth of the precision".

It is a filter by size

★★★ Big flows cannot escape at any rate; small flows almost certainly vanish. The filter is extreme, and it is invisible -- no report says "3000 flows are missing here".

★ So you get data that looks entirely normal: the byte total is nearly exact (because bytes are almost all contributed by big flows), and nobody doubts it.

★★ Meanwhile the connection count is wildly wrong

★★★ "Volumes right, counts wrong" is the signature distortion of sampled data:

· total bytes -> error may be a few percent ✓
· connection count / unique sources / unique destinations -> may be an order of magnitude low ✗

★ So any analysis based on counting (how many clients, how many connection attempts) is untrustworthy on sampled data.

Key takeaways

• ★★★ P(seen) = 1 - (1-1/N)^P. At 1:1000 a 10-packet flow has only a 1% chance of appearing.
• ★★ Sampling does not lose precision evenly -- it keeps big flows and discards small ones, invisibly.
• ★ Accurate byte totals do not mean trustworthy data: count-based analysis comes out an order of magnitude low.

③ ★★★ Why sampled data cannot find a port scan

Advanced · about 10 min

Goal: The conclusion here is subtler than "invisible": you really do see something, but what you see no longer looks like a scan.

What a scan looks like

The shape of a port scan is one source, very many targets, very few packets each.

Say 10,000 targets with 2 packets each (SYN + RST).

★ Note that any single record looks identical to a "mouse flow" -- the difference lives entirely in the aggregate shape.

Work out how much you see

At 1:1000, a 2-packet flow has a 0.2% chance of being seen.

★ So of 10,000 targets, about 20 appear in your data.

★★★ And the probability of seeing at least one is ≈100% -- a number that very easily produces a false reassurance: "look, it is in the data".

★★★ What broke is the shape, not the completeness

Here is the crux: those 20 records point at 20 unrelated targets, and read as 20 ordinary failed connection attempts -- indistinguishable from background noise.

★★★ Detecting a scan depends on density (one source hitting tens of thousands of targets in moments), and sampling is precisely what crushed the density.

★ So this cannot be fixed by "lowering the alert threshold": the shape of the evidence is already gone.

This explains a familiar argument

★★ From the same sampled data two teams reach opposite conclusions, and neither is wrong:

· the network team asks "who is using the bandwidth?" -> the answer is in the big flows -> sampling is entirely adequate
· the security team asks "is anyone scanning me?" -> the answer is in the small flows -> sampling deleted all the evidence

★★★ So the right conclusion is not "what rate should we pick" but that the two needs require two collection strategies.

Key takeaways

• ★★★ Sampling does not make a scan "invisible" -- it makes it "not look like a scan". What breaks is the shape, not the completeness.
• ★★ "At least one was sampled" is a worthless guarantee: detection needs density, and sampling destroyed the density.
• ★ Edges and critical zones need unsampled collection; sample only in the core. A strategy question, not a tuning question.

④ ★★ Your "real-time traffic dashboard" is actually lagging

Advanced · about 8 min

Goal: Understand how the flow cache and its two timeouts decide when data appears, and why alerts on traffic bursts are inherently delayed.

What happens inside the flow cache

Aggregating collection accumulates packets into a flow cache, and when it exports is decided by two timeouts:

· the active timeout (commonly 60s / 1800s): export a slice even though the flow continues
· the inactive timeout (commonly 15s): idle this long means finished

★ Plus one optimisation: seeing a TCP FIN/RST exports immediately instead of waiting out the inactive timeout.

★★★ So you can never see "now"

A download in progress, 20 seconds in, with a 60-second active timeout -- it is entirely absent from your reports right now, with 40 seconds still to wait.

★★★ What you see is always "the part of history already exported", never the present. This is not an implementation flaw; it is intrinsic to the mechanism.

★★ An easy statistical mistake

The active timeout slices a long flow into a series of records. So a one-hour download may be 60 records rather than 1.

★★★ If your analytics counts connections by number of records, it reads that one download as 60 connections.

★ The right approach is to deduplicate by 5-tuple, not count records.

What if you need second-level visibility

★ Two roads:

· shrink the active timeout (at the cost of far more records, which both the collector and the bandwidth must absorb)
· use sFlow -- no flow cache, sampled means shipped, lag ≈ 0

★★ Which is why many networks deploy both: IPFIX for exact accounting, sFlow for fast awareness.

Key takeaways

• ★★★ Aggregating collection must wait for the cache to expire, so the "live dashboard" lags by one active timeout.
• ★★ The active timeout slices long flows into series of records -- counting records as connections goes badly wrong.
• ★ Second-level visibility means shrinking the timeout (expensive) or switching to sFlow (no cache, no lag).

⑤ ★★★ The device is perfectly healthy and the UI is blank

Troubleshooting · about 10 min

Goal: Reproduce this family's most classic silent outage by hand, and take away the more general lesson: "I sent it" and "they can read it" are two different things.

Start with the symptoms

★★★ The shape of this outage is thoroughly deceptive:

· show on the device is clean
· export counters are climbing
· a packet capture proves UDP datagrams really are flying
· and the collector UI is blank

★ So people investigate the device and the network over and over -- and find nothing there.

★★★ The cause: data records are not self-describing

A NetFlow v9 / IPFIX data record contains no field descriptions, just a string of binary values. Parsing depends entirely on a template the device sent earlier.

★★ And that template is cached in the collector's memory. A process restart loses it.

★★★ The device has no idea -- it only resends templates on a fixed interval (commonly 10 minutes). Throughout those 10 minutes, every data record received can only be discarded whole.

How to fix it and how to prevent it

★ Immediate recovery: trigger a template resend on the device by hand.

★★ Long-term prevention:
· shorten the template refresh interval (it costs only a little bandwidth)
· after a collector restart, actively trigger a resend rather than waiting
· monitor the collector's parse-failure count, not just its receive count

★★★ That last one is the key: watching only "how much arrived" will never reveal this outage.

The more general lesson

★★★ This outage is worth remembering because its shape recurs everywhere:

"I sent it" does not mean "they can read it".

★ So monitoring any data path by the sender's counters alone is not enough -- you must watch how much the receiver successfully processed. The gap between those two numbers is where the problem actually lives.

Key takeaways

• ★★★ v9/IPFIX data records need a template to parse; the template lives in collector memory and vanishes on restart.
• ★★ The symptoms are thoroughly deceptive: every device-side counter is healthy while the collector discards everything it receives.
• ★ Monitor "successfully parsed", not just "received" -- otherwise this outage stays invisible forever.

⑥ Five reasons "the data is wrong", and how to tell them apart

Troubleshooting · about 10 min

Goal: Lay out the five places on the path that eat data, and learn to locate the culprit fast using three counters.

Five places, one symptom

Between real traffic and your reports, five places can make data disappear -- and they all look identical on screen:

(1) never sampled -> the data never existed
(2) still in the flow cache -> it exists, just has not arrived
(3) evicted from a full cache -> systematically loses small flows
(4) export datagram lost in transit -> produced but never delivered
(5) arrived but no template -> delivered yet unreadable

★★★ First separate "lost" from "not yet"

★★★ This is the most important fork, because the responses are opposite:

· not yet (2) -> wait one timeout and it appears; do nothing
· lost (1,3,4,5) -> waiting forever will not help

★ How to tell: wait one active timeout and look again. Still absent means lost.

★★ The three-counter method

★★ Do not guess -- compare these three numbers:

· the device's export counter -- how much it sent
· the collector's receive counter -- how much arrived
· the collector's successful-parse counter -- how much was understood

★★★ Whichever gap opens tells you the stage:
· the device count is already low -> sampling too sparse, or collection not enabled
· sent but not received -> network loss (4) or the collector cannot keep up
· received but not parsed -> template missing (5)

One closing warning

★★★ The most common wrong move when "the data is wrong" is to go change the sampling rate first.

★ Only one of the five causes (1) has anything to do with the sampling rate. Locate first, then act -- otherwise you change it, nothing improves, and you are more confused than before.

Key takeaways

• ★★★ Five places eat data between reality and reports: sampling, flow cache, cache overflow, export loss, missing template.
• ★★★ Separate "lost" from "not yet" first -- waiting helps only one of them, and the other needs no action at all.
• ★★ Locate with three counters (device export / collector receive / successful parse) instead of guessing.