The short answer
Communication Failures
A controller reports its communication problems in detail if anyone looks: each port has a status, each connection has a state, each message instruction returns an error code when it fails, and each protocol driver counts timeouts and retries. Reading those first tells you whether the controller cannot reach the medium at all, can reach it but gets no answer, or gets an answer it rejects, and each of those points at a different place: the port, cable, or switch; the device address, power, or path; or the message format, data type, or register map. The controller-side checks are cheap and they come before any site visit: the port status, the connection status, the last error code, the counters, and the configuration against the network schedule.
Key points
- Port status, connection state, and the message error code are the first three things to read.
- No link is the port, the cable, or the switch; no answer is the address, the device, or the path; a rejected answer is the message or the map.
- Controllers have connection and message rate limits; exceeding them fails the newest messages first.
- Timeouts and retries are configured per message, and set too short they cause the failures they were meant to catch.
- Compare every address, port, and rate against the network schedule before touching hardware.
Three kinds of failure
| What the controller shows | Kind | Look at |
|---|---|---|
| Port down, no link, no activity | Cannot reach the medium | Port configuration, cable, switch port, media converter, serial converter |
| Link up, messages time out | No answer | Device address, device power and state, routing, firewall, a gateway in the path |
| Answers arrive, message errors | Rejected exchange | Function code, register address and count, data type, protocol settings, unit identifier |
| Works, then fails under load | Rate or connection limit | Message count and rate, connection count, the device capacity |
| Works, fails intermittently | Marginal medium or timing | Cable, noise, timeout versus device response time, a duplicate address |
What to read
- Port status
- Link, speed, duplex, and errors on an Ethernet port; signal presence and framing errors on a serial port. The module status page or the port diagnostics.
- Connection status
- For connected protocols: the state of each connection, its error count, and the reason for the last drop.
- Message error code
- Each message instruction returns a code on failure that names the cause: timeout, connection refused, bad address, bad data, path not found. The controller manual lists them; log them in a tag so the last one is visible.
- Counters
- Timeouts, retries, and errors per message or per device, cleared and trended. A device with a rising retry count is failing before it fails.
- Configuration
- Addresses, ports, baud rate and parity, unit identifiers, timeouts, and retries, against the network schedule.
Serial links
- Baud rate, parity, data bits, and stop bits identical at both ends; a mismatch produces framing errors or silence.
- RS-485 polarity, termination at both ends of the bus only, and a bias where the driver needs it.
- One master on the bus; a second master collides with the first.
- Cable length and device count within the standard; a bus that grew past its limit fails at the far end first.
- A converter or radio in the path has its own settings and its own turnaround time; the message timeout must exceed it.
Ethernet links
- Address, mask, and gateway on the controller against the schedule; a missing gateway stops everything off the local network.
- The switch port up, at the expected speed and duplex, with no errors.
- The device reachable from the controller network; a firewall or a virtual network boundary in the path needs a rule.
- The controller connection count against its limit; every SCADA client, historian, programming session, and message consumes connections.
- The message rate against what the device can answer; a small device polled by several masters drops requests.
Timeouts and retries
A timeout shorter than the device response plus the medium latency fails good messages; a timeout too long makes a failure take minutes to notice. Set the timeout from the measured response time with margin: a local Ethernet device that answers in 10 milliseconds can have a 500 millisecond timeout; a radio path with a two second turnaround needs five seconds. Retries follow the same reasoning: one or two on a wired link, more on a radio, and never so many that the retries themselves saturate the medium. A message that fails after retries should set a communication alarm and mark its data bad, and the program should stop sending new requests to a device that is not answering until a probe succeeds.
The order of checks
- 1
Scope
One device, several, or everything on a port. Everything on a port is the port or the medium; one device is that device or its message.
- 2
Controller side
Port status, connection state, the error code, and the counters.
- 3
Configuration
Every setting against the schedule, on both ends.
- 4
Medium
Cable, switch port, converter, radio, termination.
- 5
Device
Power, address, state, and a direct test with a laptop tool.
- 6
Message
Function, address, count, and type against the device map; a test message from the laptop that mirrors the controller message.
Frequently asked questions
- The message instruction reports an error but the device answers a laptop tool.
- The controller message differs from the tool message: a different function code, a different address convention, a data type or count the device rejects, or a unit identifier. Compare the two requests byte for byte with the driver diagnostics or a capture.
- Communications fail only when the programming software is connected.
- The programming session consumed the last available connection or the controller communication time budget. Raise the time slice for communications or reduce the connections; check the controller limits.
- How do I tell a controller port failure from a switch port failure?
- Swap the cable to a known-good switch port; then swap the cable. If the link comes up on another switch port, the switch port was the fault; if it never comes up, the controller port or the cable is.
- Should the program keep polling a device that has failed?
- Not at full rate. Back off to a slow probe, alarm, and mark the data bad. Continuous retries to a dead device waste the controller communication budget and delay every other message.
Related topics
- Serial CommunicationsRS-232, RS-422, and RS-485 in practice: which is which, cable and distance, termination, biasing, the common reference wire everyone leaves off, and the settings that have to match at both ends.
- Modbus TCPModbus over Ethernet: what the MBAP header changes, why the CRC disappears, how the unit identifier works, and what does not get easier just because it is on Ethernet.
- EtherNet/IPHow EtherNet/IP moves control data: CIP over standard Ethernet, implicit I/O connections on a requested packet interval, explicit messaging, producer and consumer tags, and what the network has to provide.
- Device Times OutA polled device that answers late or not at all: how to read the driver counters, tell a dead device from a slow one, and find the timeout setting, the bus loading, the radio latency, the duplicate address, or the device that is simply too busy to answer.
- Wrong Register DataThe poll succeeds and the numbers are nonsense: an offset by one, the wrong table, swapped words in a 32-bit value, a float read as an integer, signed read as unsigned, a scale factor not applied, a stale gateway value, or the wrong unit answering.
- Network ProblemsThe controller as a network device: addressing, duplicate addresses, link speed and duplex, connection limits, packet rate and storms, multicast for remote I/O, ring protocols, and time synchronization, with the diagnostics that show which is wrong.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.