Symptom
The driver reports timeouts for a device, the tags from it go bad or stale, and the communication alarm comes and goes or stays on. Other devices on the same driver may be fine.
The short answer
Device Times Out
A device that times out either is not receiving the request, is not answering, or is answering after the master has given up. The driver counters distinguish them: no responses at all is a dead path or a wrong address; occasional timeouts with responses in between is a slow or overloaded device or a marginal path; timeouts that started after a change is the change. Measure the actual response time, compare it with the timeout setting, and work along the path from the master to the device with the tool each layer offers.
Key points
- Read the driver counters: requests, responses, timeouts, exceptions. They separate no answer from a late answer.
- A timeout shorter than the real response time is a configuration fault, not a device fault.
- On a shared bus or radio, one slow device delays every device polled after it.
- A device that answers the test tool but not the driver has a configuration difference between them.
- A duplicate address produces garbled responses and timeouts that look random.
Possible causes and what to check
| Possible cause | What to check |
|---|---|
| Timeout set shorter than the device response time | Measure the response time with the test tool or the driver diagnostics; compare with the timeout. Radio and cellular paths need seconds, not hundreds of milliseconds. |
| Device powered off, faulted, or in a mode that stops communication | Device display and status LEDs; a controller in program mode may stop serving some protocols. |
| Wrong address, unit ID, or port | Verify against the device configuration, not the drawing. A replaced device arrives with defaults. |
| Duplicate address on the bus or the network | Two devices answer; responses collide. Disconnect one and see whether the other becomes reliable. |
| Bus overloaded or poll rate too fast | Total poll time for all devices exceeds the poll interval. Slow the poll rate or split the bus. |
| Radio path marginal or shared | Retry counts and signal margin at the radios; a shared channel with other traffic. |
| Device connection limit reached | Controllers and gateways accept a limited number of TCP connections; a second master or a diagnostic tool can exhaust them. |
| Physical layer fault | RS-485 wiring, termination, bias; Ethernet cable, port errors, duplex mismatch. |
| Device too busy | A controller with a long scan or a heavy communication load answers late. Check the device scan time and its communication statistics. |
| Gateway or protocol converter stalled | The gateway answers its own diagnostics but not for the serial devices behind it. Power cycle and check its logs. |
| Firewall or access list dropping the traffic | A rule change, a new VLAN, or a device moved to a new subnet. Confirm with a packet capture at the master. |
Read the counters first
Every driver keeps statistics per device: requests sent, responses received, timeouts, exceptions, and often the last and average response time. Those numbers say which kind of failure you have before any cable is touched.
| Counters show | Meaning | Start with |
|---|---|---|
| Requests climbing, responses zero | The device never answers, or the master never sees the answer | Power, address, physical path, firewall |
| Responses present, timeouts occasional | The device answers most of the time, late or after a retry | Timeout setting, bus loading, radio margin, device busy |
| Responses present, exceptions climbing | The device answers with an error | The request itself: address range, function code, data type |
| Timeouts on every device on the driver | The driver, the port, or the network segment | The master side: port, adapter, switch, driver service |
| Timeouts on the devices after one slow device | The slow device consumes the poll cycle | The slow device and the poll schedule |
| Timeouts began at a specific time | Something changed | The change log: firmware, program, network, a new device |
Diagnostic procedure
- 1
Scope it
One device or all devices on the driver. All devices means the master, the port, or the segment; one device means the device or its branch.
- 2
Measure the response time
From the driver diagnostics or a test tool. A device that answers in 800 ms on a driver with a 500 ms timeout is not faulty; the timeout is. Radio hops add hundreds of milliseconds each; cellular adds a second or more with variation.
- 3
Test with the tool
Connect a protocol test tool to the same path, with the same address and settings, and poll. If the tool succeeds and the driver does not, compare the settings character by character: unit ID, port, function code, register range, data type. If the tool fails the same way, the path or the device is the problem.
- 4
Check the device
Power, status LEDs, communication LEDs flashing on request, display or web page reachable, mode. A controller in program mode, a device in a configuration menu, or a drive in a fault state may stop answering.
- 5
Check the address for duplicates
On a serial bus, disconnect the suspect device and see whether another device starts answering its address. On Ethernet, check the switch MAC table for the IP, or ping and look for two replies.
- 6
Check the physical layer
Serial: A and B polarity, termination at both ends only, bias at one point, shield grounding, cable near drive conductors. Ethernet: port link and errors on the switch, cable, duplex and speed, a media converter.
- 7
Check the bus loading
Add up the poll time for every device on the bus at the configured rate. If it exceeds the poll interval, the last devices in the cycle time out every cycle. Slow the rate, shorten the requests, or split the bus.
- 8
Check the connection count
On a TCP device, how many connections it allows and how many are open: the SCADA driver, a second SCADA server, a historian, a diagnostic laptop, a gateway. Close the extras.
- 9
Check the path in between
Radios: retries, signal margin, channel utilization. Gateways: their own diagnostics and logs. Firewalls: a packet capture at the master showing the request leave and no response return.
- 10
Fix, then watch the counters
After the fix, the timeout counter stops climbing and the response time settles. Watch through a full day, because loading and radio conditions change.
Common mistakes
- Replacing the device before measuring the response time.
- Shortening the timeout to make the driver report failures faster, which creates them.
- Polling every device as fast as the driver allows, then wondering why the last ones time out.
- Leaving a diagnostic laptop connected to a controller that allows two connections.
- Swapping A and B on one device of a bus and declaring the bus bad when the others stop working.
Frequently asked questions
- The device times out every few minutes and then recovers. What is it?
- A marginal path or a busy device: a radio fade, a periodic burst of other traffic, a device task that runs periodically and delays the reply, or a poll schedule that occasionally collides. The response time trend shows the peaks; match their timing to something in the system.
- The device answers the laptop but not the SCADA server. Why?
- A difference in configuration between the two masters, or a network difference: the server on a different VLAN with a firewall rule, a different unit ID, a different port, or a connection limit the server has reached and the laptop has not. Compare settings first, then the network path.
- How long should a Modbus timeout be?
- Long enough for the slowest legitimate response plus margin. On a direct serial link, 500 ms to 1 s. Through a radio, 2 to 5 s. Over cellular, 5 to 10 s. Measure and set; do not use the default.
- Can too many retries make things worse?
- Yes. Retries on a timed-out device consume poll time that the other devices needed, so one dead device slows the whole bus. Two or three retries, then mark the device offline and poll it less often until it answers.
Related topics
- Modbus Device Intermittently OfflineA device that answers most of the time. Termination, bias, timing, and the physical layer details that produce a bus which almost works.
- Ethernet Device Drops OfflineAn industrial Ethernet device that disappears and comes back. Duplex mismatch, cabling, spanning tree, and the switch counters that identify it in minutes.
- How to Configure a Modbus ConnectionSet up a Modbus RTU or TCP link between a controller and a device: get the register map, match the physical and protocol settings, build the reads and writes, handle data types and the addressing offset, and verify with a protocol tool before trusting a value.
- Serial CommunicationsRS-232, RS-422, and RS-485 in practice: which is which, cable and distance, termination, biasing, the common reference wire everyone leaves off, and the settings that have to match at both ends.
- Modbus TCPModbus over Ethernet: what the MBAP header changes, why the CRC disappears, how the unit identifier works, and what does not get easier just because it is on Ethernet.
- How to Diagnose a Bad Quality TagTrace a bad quality indication from the SCADA screen back to its source: the tag configuration, the driver and device connection, the controller tag, and the field signal, using the diagnostics each layer provides, and separate a stale value from a bad one.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.