The short answer
How to Diagnose Packet Loss
To diagnose packet loss, confirm it with a continuous timestamped ping from a station near the affected device, then ping each hop along the path in turn to find the first segment that loses packets, ping from the far end back to confirm the segment, clear and read the error and drop counters on every switch port in that segment, and match the pattern of loss to its cause: random loss with CRC errors is a cable or noise, loss under load is congestion or a duplex mismatch, periodic bursts are a device or a scheduled job, brief total loss is a link flap or spanning tree, and loss behind a radio is the radio link.
Key points
- Measure before you touch anything: a timestamped continuous ping gives the loss rate and the pattern.
- Localize by hop: the first hop that loses is where the problem is, and the far-end ping confirms it.
- Port counters name the layer: CRC errors are physical, output drops are congestion, both zero means the loss is beyond the switch.
- The pattern is the diagnosis. Random, under load, periodic, and burst each have a short list of causes.
- A controller that drops pings under load is often the controller, not the network; the protocol timeouts are the real measure.
- Zero loss over an hour, with counters not moving, is the finish line.
What you need
- A laptop that can be plugged into the switches along the path
- The network drawing with every switch, link, and radio on the path
- Management access to every switch on the path and to any radio or cellular modem
- The controller or SCADA communication status and timeout counters
- A spare patch cable and, for fiber, a light meter
- A text file or notebook for the ping logs and counter readings
Procedure
- 1
Confirm and measure the loss
From a station as close to the affected device as practical, run a continuous ping with timestamps for at least fifteen minutes and save the output. Note the loss percentage and the pattern: single random drops, bursts, drops at regular intervals, or long outages.
- 2
Walk the path on the drawing
List every hop between the client and the device: switches, trunks, fiber links, radios, the firewall. Loss on a path can only come from a link, a switch, or the end device, so the list is the list of suspects.
- 3
Ping hop by hop
From the same station, ping the management address of each switch along the path in order, then the far device. The first target that shows loss marks the segment: the loss is between the last clean target and the first lossy one. If every switch is clean and only the device loses, it is the device, its cable, or its port.
- 4
Ping from the far end
Plug the laptop in beside the affected device and ping back toward the client. Loss that appears in both directions on the same segment confirms the segment. Loss in one direction only is often a duplex mismatch or a half-broken pair in one cable.
- 5
Read the counters on the segment
On both ports of the suspect link, clear the counters, wait ten minutes, and read input errors, CRC errors, output drops, and collisions. CRC and input errors are physical: cable, connector, transceiver, noise. Output drops are congestion. Late collisions are a duplex mismatch. Clean counters on a lossy segment point at a radio, a loop elsewhere flooding the VLAN, or the far device.
- 6
Check the radio or cellular link
If the segment includes a radio, read its received signal strength, signal-to-noise ratio, and error or retry counters on both ends, and compare with the values recorded at commissioning. Loss on a radio link that tracks weather, time of day, or a new obstruction is the link, not the network.
- 7
Check for a loop or a storm
Loss across a whole VLAN at once, switches with high CPU, and every port light solid are a loop or a broadcast storm. Read the switch logs for topology changes and the storm control counters, and find the new cable.
- 8
Check the end device
A controller that loses pings only when it is busy, and whose protocol connections stay good, is deprioritizing ICMP. A controller whose protocol connections also time out under load has too many clients or too fast a poll; count the connections and slow the polls.
- 9
Fix the cause, not the symptom
Replace the cable or transceiver, correct the duplex, enable IGMP snooping for multicast flooding, enable spanning tree and storm control for loops, realign or re-aim the antenna, isolate the chatty device, or spread the polls. Then repeat the measurement.
- 10
Verify and record
Run the continuous ping again for an hour with the counters cleared. Save the before and after ping logs and counter readings with the work order so the next person has a baseline.
Loss patterns and their causes
| Pattern | Likely cause | Confirming evidence |
|---|---|---|
| Random single drops, 1 to 5 percent | Cable, connector, transceiver, or noise | CRC and input errors climbing on one port; loss tracks a drive or a motor running |
| Loss only under load | Duplex mismatch or congestion | Late collisions on one side; output drops on a trunk; loss disappears when polling is paused |
| Drops at a regular interval | A scheduled job or a device with a periodic burst | Timestamps match a backup, a scan, a historian job, or a device that floods multicast on a timer |
| Complete loss for 10 to 60 seconds, then recovery | Link flap or spanning tree reconvergence | Switch log shows the port going down and up or a topology change at the same time |
| Loss to everything behind one link | That link: fiber, radio, cellular, or the trunk | Both ends of the link show it; devices on the near side are clean |
| Loss to one device only | Its cable, port, or the device itself | Every switch on the path is clean; counters on its port or none at all |
| Loss across a whole VLAN | Loop, broadcast storm, or multicast flood | Switch CPU high, storm control counters, all port lights solid, IGMP snooping off |
Measuring
REM Windows: continuous ping, then per-hop loss
ping 10.5.10.21 -t
pathping 10.5.10.21
# Linux: 1000 pings at 200 ms, then per-hop loss
ping -c 1000 -i 0.2 10.5.10.21
mtr -r -c 200 10.5.10.21
# Switch: clear, wait, read
clear counters GigabitEthernet1/0/24
show interfaces GigabitEthernet1/0/24 | include error|drops|collisionA ping with a large payload, near 1400 bytes, finds problems that a default 32-byte ping misses: a marginal cable drops long frames first, and a path with a mismatched maximum frame size drops them entirely. Run both sizes when the default ping looks clean and the protocol still times out.
Verification
- A one-hour continuous ping shows zero loss on wired paths, or loss within the commissioning baseline on radio and cellular paths.
- CRC, input error, output drop, and collision counters on every port in the path stay at zero over the same hour.
- The controller and SCADA communication status stays good, and the timeout counters stop incrementing.
- The switch logs show no link flaps or topology changes during the test.
- The before and after measurements and the cause are recorded with the work order.
Frequently asked questions
- The switch counters are all zero but ping still loses packets. Where is the loss?
- Somewhere the counters do not see: a radio link that retries silently, a firewall that rate-limits ICMP, an unmanaged switch in the path, or the end device dropping pings under load. Ping the far side of each of those in turn, and test with the protocol rather than ping if the end device is the suspect.
- How much loss is acceptable?
- On a wired control network, none; a switched Ethernet path with good cables delivers every frame. On radio and cellular, a percent or two is normal and the protocol timeouts and retries are designed for it. Anything above the commissioning baseline on any link is a fault to find.
- Ping loss started after a firmware update on the controller. What changed?
- Probably the priority the controller gives to ICMP, or the number of connections it accepts. Check the protocol communication status first; if it is good, the loss is a measurement artifact. If the protocol also fails, the update may have changed connection limits or the default port speed and duplex.
- Can a bad cable on one device cause loss on other devices?
- On a switched network, a bad cable affects only its own port, unless the errors are severe enough to flood the switch with malformed frames or the cable is a trunk. Loss on several devices at once points at a shared link, a loop, or a storm, not at one device cable.
Related topics
- How to Troubleshoot an Ethernet ConnectionWork an Ethernet problem one layer at a time: link light and cable, then speed, duplex, and VLAN on the switch port, then address, mask, and gateway, then the application port through the firewall. Each layer has a one-minute test that rules it in or out.
- Ethernet Device Drops OfflineAn industrial Ethernet device that disappears and comes back. Duplex mismatch, cabling, spanning tree, and the switch counters that identify it in minutes.
- Device Times OutA polled device that answers late or not at all: how to read the driver counters, tell a dead device from a slow one, and find the timeout setting, the bus loading, the radio latency, the duplicate address, or the device that is simply too busy to answer.
- Modbus Device Intermittently OfflineA device that answers most of the time. Termination, bias, timing, and the physical layer details that produce a bus which almost works.
- How to Diagnose a Bad Quality TagTrace a bad quality indication from the SCADA screen back to its source: the tag configuration, the driver and device connection, the controller tag, and the field signal, using the diagnostics each layer provides, and separate a stale value from a bad one.
- How to Assign IP Addresses on a Control NetworkBuild an addressing plan before the first device is configured: one subnet per zone and site, a fixed block layout so an address says what the device is, static addresses on everything that controls a process, and a schedule that is kept current.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.