Lessons (6, study in order)

① A log is not a complete record

Beginner · about 8 min

Goal: Correct one intuition first: absent from the logs does not mean it did not happen.

A log line has to survive four gates

★★★ Checking the logs after an incident and treating absence as proof is wrong. A line has to pass four gates:

· the level filter: anything less severe than the device's logging level is never sent;
· the device rate limit: excess lines are silently merged or dropped;
· device load: at full CPU the logging process cannot get scheduled;
· transport: UDP has no retransmission, and a full collector cannot keep up.

★★★★ The damning part is that all four correlate with the fault

During an incident there are more lines (so the rate limit trips), higher CPU (so nothing gets sent) and a busier link (so UDP drops more).

★★★★ Which makes the module's sentence precise: logs are most reliable exactly when you do not need them.

★★ This is not syslog being badly built; it follows from a connectionless, stateless, best-effort design.

★ Start on the baseline and see what correct looks like

This scenario has no rate limit, no congestion and synchronised clocks. However many times something happened, that many lines are stored.

★ Every one of the next five lessons is measured against it.

Key takeaways

• Absent from the logs does not mean it did not happen
• Four gates: level, rate limit, load, transport
• ★★★ All four correlate with the fault, so logs fail when you need them

② ★★★ The line you lose is the only useful one

Advanced · about 10 min

Goal: See how UDP drops under a log storm, and why it drops the line that mattered.

The scenario: one flapping link starts a storm

A link is flapping, so 180-odd lines pour out. The actual root cause is a single power-supply alarm on the core router -- one line, in the same second as the storm (the power fault is what causes the flapping, so they cannot be apart).

★★★★ Collector capacity is shared globally per second

★★★ Not a quota per event -- within one second it is first come, first served (which is how a UDP receive buffer behaves).

★★★★ So the storm's 120 lines fill every slot and the power alarm gets none. It was not "a few also lost" -- it was erased completely.

★★ And neither end has a record: the device believes it sent it, the collector never knew it existed.

★★ Does TCP fix this

It buys retransmission at the cost of back-pressure: when the collector is slow the device's send buffer fills, and then it either blocks the process producing logs or drops anyway.

★★★ So TCP only converts "dropped" into "dropped later, and possibly dragging the device down" -- not free reliability. The real remedy is to throttle the storm source on the device and give critical severities their own path.

Key takeaways

• ★★★★ Collector capacity is global per second, first come first served
• A log storm crowds out the only useful line
• TCP's price is back-pressure, not free reliability

③ ★★★★ One line does not mean one occurrence

Troubleshooting · about 8 min

Goal: Meet syslog's most deceptive loss: the line was not missing, it was merged away.

Devices rate-limit to protect themselves

logging rate-limit is necessary: without it one storm can exhaust a device's CPU.

★★ But the excess is discarded silently -- the device does not log a line saying "I just dropped four".

★★★★ Why this loss deceives

★★★★ Because the line is in the collector. Nobody suspects loss -- yet "one line" reads as "one occurrence".

The link flapped five times, the on-call engineer sees a single blip, files it as intermittent and moves on. Five flaps mean a hardware fault that is getting worse.

★★★ The remedy: do not infer counts from line counts

★★★ Read the interface's own counters on the device (flap counter, input errors) -- the syslog rate limit does not touch those.

★★ The discipline generalises: never count occurrences from log lines. Logs answer "what happened"; they do not answer "how many times".

Key takeaways

• ★★★★ Rate limiting is silent: the line is there, the count is not
• Count from the device's own counters
• Logs answer what happened, not how often

④ ★★★★ The order you sorted by may be wrong

Troubleshooting · about 10 min

Goal: Understand why sorting by timestamp sends you to the wrong end.

Troubleshooting almost always starts with sorting

Line up several devices' logs by time and start from the earliest, because the thing that happened first is usually the root cause.

★★★★ But a syslog timestamp is stamped by the device's own clock, not by the collector.

★★★★ Once the clocks differ by more than the gap, the order flips

In this scenario D1 fails at second 10 (the cause) and D2 is affected at second 12 (the effect). D1's clock is four seconds fast, so the logs read 14 and 12.

★★★★ Sorted by timestamp, the effect comes before the cause. You start on D2, which really is healthy -- you look, find nothing, and begin suspecting an intermittent fault.

★★★ This lesson follows the time-sync page

NTP and PTP cover why clock sync is hard: software-stamp jitter, queueing, path asymmetry.

★★★★ This lesson covers what it costs when it fails -- and the cost lands precisely on the action troubleshooting depends on most.

★★ Note a log has no way to notice its own timestamp is wrong. It is written in the correct format and looks entirely normal.

★★★ The remedy: never build causality from log timestamps alone

Cross-check three things:

· the interface's own counters on the device;
· the collector's receive time, which is stamped by one clock and is therefore comparable;
· the routing protocol's own convergence events.

Key takeaways

• ★★★★ The device stamps the time, so a clock offset flips the order
• Syslog's usefulness depends on NTP
• The collector's receive time comes from one clock and is comparable

⑤ Severity, format, and two common traps

Beginner · about 9 min

Goal: Settle severity and message format in one go.

★★★ A lower severity number means more severe

0 is emerg, 7 is debug. ★★ That alone is counter-intuitive -- plenty of people write their first alert rule as severity > 4 and receive nothing but debug.

★★★ So logging level 4 means "send 0 through 4 only".

★★★★ More importantly, the vendor assigns that number

★★★★ And vendors disagree. The same condition is warning(4) on one platform and notice(5) on another.

★★★ Alert rules are written against the number, so one rule fires on one platform and stays silent on the other. In this scenario "switched to the backup PSU" is tagged notice(5) and level 4 blocks it.

★★ And it cannot be found on the device either: never sent, so absent from the device's counters too.

★★ Format: RFC 3164's timestamp is missing three things

· no year -> ordering breaks across a year boundary;
· no timezone -> logs from different regions cannot be compared directly;
· second precision only -> the order of events within one second is simply lost.

★★ RFC 5424 and JSON supply all three. ★★★ But the loss problem is completely unchanged -- format fixes legibility, not reliability.

Key takeaways

• ★★★ Lower severity numbers are more severe, and vendors assign them
• A level set too high hides the event from the device too
• ★★★ Format fixes legibility, not reliability

⑥ ★★★ The logging path must not depend on what it watches

Troubleshooting · about 9 min

Goal: Connect syslog back to a larger principle.

The scenario: collector and devices share an uplink

To save a cable the collector hangs off the same access switch. Fine day to day.

★★★★ When that link fails you lose sight of the device and of why it went -- at exactly the moment you need the logs most.

★★★★ This is the same principle again

★★★★ The three-networks page states it as: a management path must not depend on the thing it manages.

★★★ This lesson only swaps "management path" for "logging path". Every "we could not diagnose it remotely and had to drive over" has the same root: the observation path shared a segment with the thing that failed.

★★ What to actually do

· put the collector on the out-of-band management network, not the production uplink;
· configure two syslog targets on critical devices, in different fault domains;
· keep a local buffer on the device (logging buffered) -- it is the only way to recover what happened while the link was down;
· ★★★ monitor the collector itself: the absence of incoming logs must be alertable on its own.

★★ To close: how the four observability tools divide the work

★★ The four modules in this category each watch one thing:

· traffic (NetFlow, IPFIX) -- who is talking to whom;
· metrics (SNMP, gNMI) -- how a number moves over time;
· routes (BMP) -- why a route is not in my table;
· events (syslog) -- what happened.

★★★ All four degrade during a fault, and each degrades differently -- which is why no single one of them is enough.

Key takeaways

• ★★★★ An observation path must not share a segment with the fault
• Two syslog targets plus a local buffer on critical devices
• ★★★ The collector receiving nothing must itself raise an alert