Symptom
Values from devices behind a gateway are frozen, zero, or bad quality while the gateway itself pings and answers polls; the failure appeared after a power event, a firmware update, a configuration change, an added device, or an increase in polling; all devices behind the gateway are affected, or only some.
The short answer
Protocol Gateway Stops Passing Data
A protocol gateway has two sides, and a gateway that stops passing data has usually failed on one of them while the other keeps working, which is why the master still sees it answering. The device side can lose a serial link, a device, or a port; the master side can be overloaded by polls or by too many connections; and the gateway itself can stall, lose its configuration on a power cycle, fill an internal buffer, or freeze its mapping so that it serves stale values forever. The test is to read each side independently: poll the device directly if the medium allows, read the gateway diagnostics for the device-side status, and read the gateway with a test master to see whether its table is live. A gateway that serves the last good value instead of a fault when its device side dies is the worst case, because everything looks fine until someone notices the numbers have not moved.
Key points
- Test the two sides separately: the device side status in the gateway diagnostics, and the master side with a test master.
- A gateway that answers with stale values on a dead device side hides the fault; configure it to report bad quality instead.
- Power cycles lose unsaved configuration, and a gateway running defaults answers with nothing useful.
- All devices frozen is the gateway or its device port; some devices frozen is those devices or the wiring to them.
- Too many masters or too fast a poll starves the device side, and the gateway serves what it has.
Possible causes and what to check
| Possible cause | What to check |
|---|---|
| Device-side link failed | The gateway diagnostics show timeouts or errors on the serial or device port for every device; a broken cable, a failed converter, a dead device port, or a termination problem. Fix the device side. |
| One device failed or was removed | Only that device is frozen; its status in the gateway shows timeouts. The device, its power, or its address. |
| Configuration lost on power cycle | The gateway came back on defaults: wrong device addresses, no mapping, wrong baud rate. Restore from the saved configuration and save to non-volatile memory. |
| Stale value hold | The gateway holds the last good value when a device stops answering and reports it as good. Configure the gateway to zero the value or set a fault flag on timeout, and alarm on it. |
| Gateway stalled or hung | Pings answer but polls time out or return the same values indefinitely; a firmware fault or a full buffer. Power cycle, then update firmware if it recurs. |
| Master side overloaded | Several masters polling, or one polling faster than the device side can be served; the gateway answers with what it has and the device side falls behind. Reduce masters and poll rate. |
| Connection limit reached | A master cannot connect while another holds all the sessions; abandoned connections from a previous master. Set a session timeout and limit the masters. |
| Mapping error after a change | A device added or an address changed without the mapping updated; values appear in the wrong place or not at all. Review the mapping against the change. |
| Firmware update changed behavior | Defaults, addressing, or word order changed with the update; the master sees wrong or missing data. Read the release notes and reconfigure. |
| Time or watchdog reset loop | The gateway reboots on a schedule or a fault and loses data during each reboot; the uptime counter. Fix the cause of the resets. |
Two sides, two tests
| Test | Result | Meaning |
|---|---|---|
| Ping the gateway | Answers | The network side and the gateway processor are alive; nothing more |
| Gateway diagnostic page | Device side shows timeouts or errors | The device side: cable, device, port settings, termination |
| Gateway diagnostic page | Device side shows good polls | The devices are answering; the fault is in the mapping or the master side |
| Test master reads the gateway | Values live and changing | The gateway is serving; the fault is in the SCADA driver or its configuration |
| Test master reads the gateway | Values frozen | The gateway table is stale: a stalled gateway, a dead device side with hold enabled, or a mapping that stopped |
| Direct poll of a device, bypassing the gateway | Answers | The device is fine; the gateway or its port is not |
Working it
- 1
Scope
All devices behind the gateway, or some. All is the gateway or its device port; some is those devices.
- 2
Diagnostics
The gateway web page or console: uptime, device-side status per device, error counters, connection list, and the configuration version. A short uptime means it rebooted; a default configuration means it lost its settings.
- 3
Device side
Serial: the converter, the cable, the termination, the baud rate and parity, and each device address. Ethernet devices behind a gateway: ping and poll them directly.
- 4
Master side
How many masters, how fast, and how many connections are open; compare with the gateway limits.
- 5
Mapping
The mapping table against the current device list and addresses, especially after any change.
- 6
Restart and restore
If the gateway is hung, restart it; if it came back on defaults, restore the saved configuration and save it to non-volatile memory. Then update firmware if the stall recurs.
- 7
Fix the hold
Configure the gateway to report a fault on device timeout rather than the last value, and add an alarm on the fault flag or a heartbeat register.
Serial device side
Most gateways carry a serial multidrop on the device side, and most device-side failures are the serial ones: a termination that was removed with a device, a converter that failed, a cable damaged during other work, a device set to the wrong baud rate after a replacement, or two devices with the same address after a swap. The gateway error counters per device separate a failed bus, where every device times out, from a failed device, where one does. A test master on the serial bus, with the gateway disconnected, proves the bus without the gateway in the way.
Designing it not to happen
- Save the configuration to non-volatile memory and keep a backup file with the network schedule.
- Report faults, not held values, on device timeout, and alarm on them.
- Map a heartbeat register from each critical device through the gateway.
- One master, or as few as the design needs, with a poll rate the device side can serve with margin.
- A configuration record that includes firmware version, so an update is a planned change with release notes read.
- A spare gateway configured and on the shelf; a gateway is a single point of failure for everything behind it.
Frequently asked questions
- The gateway pings and answers Modbus but every value is zero.
- The gateway lost its device-side link or its configuration and is serving an empty table. The diagnostics page shows which; a gateway on defaults after a power cycle is the usual finding, and the saved configuration is restored and written to non-volatile memory.
- Data stops every few hours and comes back after a minute.
- The gateway is rebooting: a watchdog reset from a firmware fault, a power supply that dips, or a scheduled reset. The uptime counter proves it, and the fix is the power supply or a firmware update.
- Adding one more device made everything slow and then stop.
- The device side poll cycle grew beyond what the master side expects, the master started timing out and retrying, and the retries made the cycle longer. Slow the master, poll fewer registers, or split the devices across two gateways.
- Why does the SCADA say the data is good when it is frozen?
- Because the gateway answered the poll with a valid frame containing the held value, and the driver cannot tell a held value from a fresh one. Bad quality has to come from the gateway reporting the timeout, or from a heartbeat the controller checks.
Related topics
- Protocol GatewaysProtocol converters between a controller and devices that do not speak its language: what a gateway does, the common pairings in water and wastewater, the mapping table that is the whole configuration, the failure modes a gateway adds, and when to use a native interface instead.
- Device Times OutA polled device that answers late or not at all: how to read the driver counters, tell a dead device from a slow one, and find the timeout setting, the bus loading, the radio latency, the duplicate address, or the device that is simply too busy to answer.
- Modbus Device Intermittently OfflineA device that answers most of the time. Termination, bias, timing, and the physical layer details that produce a bus which almost works.
- Wrong Register DataThe poll succeeds and the numbers are nonsense: an offset by one, the wrong table, swapped words in a 32-bit value, a float read as an integer, signed read as unsigned, a scale factor not applied, a stale gateway value, or the wrong unit answering.
- Values Frozen On ScreenThe HMI shows numbers that are not changing while the plant runs. Whether the controller, the communication driver, the tag server, or the display stopped, and the order to check them in.
- Serial CommunicationsRS-232, RS-422, and RS-485 in practice: which is which, cable and distance, termination, biasing, the common reference wire everyone leaves off, and the settings that have to match at both ends.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.