Symptom
Occasional poll failures and retries on an otherwise working network; tags flicker to bad quality for a scan and return; a continuous ping shows a small percentage of loss; the loss is worse at certain times, under load, or to certain devices; nothing is fully down.
The short answer
Intermittent Packet Loss
Packet loss on a plant network is almost never random. It comes from a specific link or device that damages, drops, or delays frames under a specific condition: a marginal cable or a dirty fiber connector that corrupts frames, a duplex mismatch that causes collisions and late frames under load, a link or a switch port saturated by a burst so that the buffer overflows, a device with a small processor that drops requests when polled too fast, or a layer two loop that floods the network intermittently. The port error counters on the managed switches and a loss test run hop by hop find the link; the pattern in time, whether the loss follows load, follows a device, or follows the weather, names the cause.
Key points
- Read the error counters on every port in the path before doing anything else; a counter that grows names the link.
- Loss that follows load is congestion or a duplex mismatch; loss that follows one device is that device; loss that follows the weather is a radio or fiber hop.
- Cyclic redundancy check errors mean a physical problem; discards mean a buffer or bandwidth problem.
- Slow devices drop requests when polled faster than they can answer; that is loss too.
- Test hop by hop from the polling host outward; the first hop with loss is the fault.
Possible causes and what to check
| Possible cause | What to check |
|---|---|
| Bad cable, connector, or port | Cyclic redundancy check and alignment errors growing on one port; a cable pinched, too long, crimped badly, or run beside power. Replace the cable, then test the port. |
| Duplex mismatch | Late collisions and errors on a link with one end forced and the other autonegotiating; loss rises steeply with load. Set both ends the same. |
| Link or port saturated | Output discards on a port during bursts; a historian backfill, a camera, or a file copy sharing a control link. Separate the traffic or upgrade the link. |
| Small switch buffers and bursty traffic | Discards without high average utilization; many devices answering a broadcast poll at once. Stagger the polls or use a switch with deeper buffers. |
| Marginal radio or fiber hop | Loss that follows weather or time of day; the radio retry count or the fiber receive power. Fix the hop. |
| Device cannot keep up | Loss only to one device, with the switch counters clean; a gateway or an old controller polled by several masters. Slow the poll rate or reduce the masters. |
| Intermittent loop or a flapping redundant path | Bursts of loss across the network with broadcast traffic spikes; a ring or spanning tree reconverging. Find the topology event. |
| Unmanaged switch or hub in the path | A daisy chain through an unmanaged device with no counters, where a bad port hides. Replace it with a managed switch or bypass it to test. |
| Wireless bridge in the path | Loss that follows interference or a moving obstruction; the bridge statistics. Treat it as a radio path. |
| Address conflict | Loss to one address only, with replies from two different hardware addresses; find the duplicate. |
Read the counters first
Every managed switch counts what happens on each port: frames received with a bad check, frames that were too long or too short, collisions, and frames discarded because a buffer was full. A plant network with loss has a port somewhere whose counters are climbing, and reading them is faster than any test. Clear the counters, wait through a period of loss, and read them again; the port with the growth is the link to look at.
| Counter growing | Meaning | Look at |
|---|---|---|
| Cyclic redundancy check or alignment errors | Frames damaged in transit: a physical fault | The cable, the connectors, the port, noise coupling from power |
| Late collisions | A duplex mismatch on this link | The speed and duplex settings at both ends |
| Runts and giants | Malformed frames: a bad transceiver, a mismatch, or a faulty device | The device on the port and the media |
| Output discards | The port could not send fast enough: congestion | What is sharing this link; bursts; the link speed |
| Input discards | The switch could not process frames: overload or a storm | Broadcast levels, loops, the switch load |
| Link transitions | The port went down and up | The intermittent link article for this media |
Find the hop
- 1
Baseline the loss
A continuous ping from the polling host to the affected device at a short interval for an hour, logging the results, so that there is a number and a time pattern to compare against.
- 2
Walk the path
From the network drawing, list every switch and link between the host and the device, including any media converters, radios, or unmanaged switches.
- 3
Ping each hop
From the host, ping the management address of each switch in order. The first hop that shows loss is where the fault begins; everything before it is clean.
- 4
Read that link
The port counters at both ends of the losing link and the settings, then the media: cable test, fiber power, or radio statistics.
- 5
If every hop is clean
The loss is at the device: poll rate against what it can answer, its processor load, its own port counters, or a duplicate address.
- 6
Fix and retest
Repeat the baseline ping after the fix and keep it running; loss that had one cause often had two.
Loss that follows load
Control traffic is small and regular, and it lives happily on a link until something large joins it: a historian backfilling after an outage, a camera stream, a file transfer, a backup, a workstation update. The output discards on the shared port climb during those periods and the polls that lose are the ones that arrived while the buffer was full. The cures are separation, by putting the large traffic on its own virtual network or link, quality of service that gives control frames priority, and where the link is a slow one, a faster link. A gigabit link that is saturated by control traffic alone has a loop or a storm, not a capacity problem.
Loss that follows a device
A device that drops requests is at its limit. Old controllers with serial-era processors, protocol gateways translating for several masters, and cellular modems with small buffers all have a request rate beyond which they drop or delay. The polling host sees timeouts; the network sees nothing. The setting is the scan rate for that device, the number of masters that talk to it, and the size of the requests, and the measurement is the device response time as the poll rate is varied.
Frequently asked questions
- Is 1 percent loss acceptable on a control network?
- No. A healthy wired plant network loses nothing; a fraction of a percent is a fault that is going to grow. Radio and cellular paths have loss by nature, and the polling design there tolerates it with retries and timeouts sized for the medium.
- The counters are all zero but the polls still fail.
- Counters clear on a reboot and some switches count only what they are configured to; confirm the counters are live by finding one that moves. If they truly are clean, the loss is in a device rather than a link: the polled device, the polling host, or a firewall between them dropping what it does not recognize.
- Could the loss be a security appliance?
- Yes: a firewall with a session table that fills, a deep packet inspection engine that drops a frame it cannot parse, or an intrusion prevention rule that triggers on a legitimate protocol. The firewall log shows the drops with a reason.
- How fast should I poll?
- As slowly as the process allows and no faster than the slowest device in the poll can answer with margin. A device that takes 40 milliseconds to answer cannot be polled at 50 by three masters. Measure the response time and set the scan rate from it.
Related topics
- How to Diagnose Packet LossFind where and why frames are dropped: measure loss with a continuous ping, localize it hop by hop from both directions, read the port counters on the path, and match the pattern to its cause: a cable, a duplex mismatch, congestion, a loop, or a radio link.
- Switch Port Errors IncrementingWhat each counter on a managed switch port means and which fault makes it grow: check errors from a damaged cable or noise, late collisions from a duplex mismatch, runts from a bad transceiver, discards from congestion, link flaps from a marginal connection.
- Ethernet Device Drops OfflineAn industrial Ethernet device that disappears and comes back. Duplex mismatch, cabling, spanning tree, and the switch counters that identify it in minutes.
- Broadcast StormA network suddenly slow or dead everywhere, every link light solid: a loop between switches without protection, a failed ring protocol, or a device flooding broadcasts. Recognising a storm, breaking it with a cable pull, finding the loop, and preventing it.
- Modbus Device Intermittently OfflineA device that answers most of the time. Termination, bias, timing, and the physical layer details that produce a bus which almost works.
- Tag Shows Bad QualityA tag flagged bad or uncertain by the driver. What the quality codes mean, the difference between a device that is not answering and an address that does not exist, and how to clear each.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.