The short answer
Network Problems
A controller on an Ethernet network can fail as a network device in ways that have nothing to do with its program: a wrong address or mask, a duplicate address, a port negotiating to the wrong speed, a connection table that is full, a packet rate that overloads its network interface, a broadcast storm from a loop elsewhere, or a ring protocol that has not converged. The controller web page or diagnostic display reports most of these, the managed switch reports the rest, and between them the problem is located without a program change. The same diagnostics show when the controller is the cause of the problem for everyone else, which happens when its remote I/O multicast is flooding a network that does not filter it, or when too many clients are polling it.
Key points
- Read the controller network diagnostics: address, link, connections in use, packet rate, and errors.
- A duplicate address takes a controller off the network intermittently; the switch address table finds the other device.
- Every SCADA client, historian, programming session, and messaging peer uses connections; the table fills.
- Packet rate above what the interface can process drops I/O connections first.
- Remote I/O multicast, ring protocols, and time synchronization each need the switches configured to match.
Where to look
| Symptom | Controller diagnostic | Switch diagnostic | Usual cause |
|---|---|---|---|
| Controller unreachable | Link state, address | Port state, address table | Wrong address or mask, cable, port down |
| Intermittently unreachable | Duplicate address warning, link flaps | Address learned on two ports | Duplicate address or a marginal cable |
| Clients refused | Connections in use at maximum | Nothing | Connection limit reached |
| I/O connections dropping | Packet rate, missed packets | Port utilization, discards | Overloaded interface or congested link |
| Everything slow at once | Packet rate very high | Broadcast rate, processor load | Broadcast storm from a loop |
| Remote I/O flaky through a switch | Connection timeouts | Multicast flooding or blocked | Multicast filtering not configured |
| Ring does not heal | Ring status, fault location | Ring protocol status | Ring protocol misconfigured or a second break |
| Timestamps wrong | Time source status | Time protocol status | No time source, or a boundary blocking it |
Addressing
A controller with a static address and no default gateway works on its own network and answers nothing beyond it, which is fine until a historian on another network needs it. A wrong mask makes it treat remote addresses as local, so it never sends them to the gateway. A duplicate address, from a laptop, a replacement device, or a cloned configuration, produces intermittent loss as the switches learn one device and then the other. The controller address, mask, and gateway are on the network schedule; the switch address table shows where each hardware address lives.
Connections and rate
Every controller has a limit on simultaneous connections and on packets per second through its interface. The limit is shared by SCADA clients, historians, programming sessions, messaging between controllers, and remote I/O. When the connection table is full, the next client is refused; when the packet rate exceeds the interface capacity, I/O connections time out first because they are the most time sensitive. The controller diagnostics show connections in use and the packet rate, and the design keeps both below about three quarters of the limit. The usual growth is a second SCADA server, a historian added later, and a few laptops left connected.
Remote I/O and multicast
Some remote I/O protocols use multicast for input data. On a network of unmanaged switches every port receives the multicast, which works until the traffic grows; on managed switches the multicast is either filtered with a group management protocol and a querier, or flooded. A switch configured to filter without a querier drops the traffic after a timeout, and the I/O connection fails a few minutes after every power cycle. The switches on an I/O network are configured to match the protocol, and unicast is used where the protocol offers it.
Rings and redundancy
A ring of switches or devices heals a single break only when every device on the ring runs the same protocol with the same settings and one device is the supervisor. A device in the ring that does not participate breaks the protocol and turns the ring into a loop when the break is repaired, which is the broadcast storm. The controller diagnostics show the ring state and, for device-level rings, the location of a fault; the ring should be tested by breaking it during commissioning, and the fault alarmed, because a ring with an undetected break is a line waiting for its second break.
Time
Controllers stamp alarms and events, and the stamps are only useful if the clock is right. A controller with no time source drifts minutes per month; one set by SCADA has whatever error the write path adds; one synchronized by a network time protocol is within milliseconds. Whichever is used, the source and its status are on the network schedule, the controller alarm on loss of synchronization is enabled, and the boundary firewall passes the time protocol.
Frequently asked questions
- The controller web page says the address is right but nothing can reach it.
- The mask or the gateway, the switch port, or a firewall between. Ping from a device on the same switch first; if that works, the problem is beyond the local network, and it is the mask, the gateway, or a rule.
- I/O drops every few minutes after a switch was replaced.
- The new switch is filtering multicast without a querier, or is not configured for the ring the old one was in. Configure the switch to match the I/O protocol and the ring.
- How many connections is too many?
- Above about three quarters of the controller limit at normal operation, because failover, maintenance sessions, and retries need the rest. The limit is in the controller specifications and the current count is on its diagnostics page.
- Can the controller cause a broadcast storm?
- It can cause a multicast flood on an unfiltered network, and a controller with two ports bridged internally can close a loop if both are cabled to the same switch without ring protection. A loop between switches is the usual storm source; the controller is a rarer one.
Related topics
- Duplicate IP AddressTwo devices claiming one address: a controller that answers sometimes, a workstation that reports a conflict, a ping returning from two hardware addresses. Where duplicates come from, finding both in the switch address table, and the discipline that stops it.
- Broadcast StormA network suddenly slow or dead everywhere, every link light solid: a loop between switches without protection, a failed ring protocol, or a device flooding broadcasts. Recognising a storm, breaking it with a cable pull, finding the loop, and preventing it.
- Ethernet Device Drops OfflineAn industrial Ethernet device that disappears and comes back. Duplex mismatch, cabling, spanning tree, and the switch counters that identify it in minutes.
- Remote I/OExtending controller I/O to another panel or another site over a network: the adapter and the rack, the update rate and what it costs, what the outputs do when the link fails, the difference between remote I/O and a remote controller, and the wiring, addressing, and diagnostics that make it dependable.
- How to Assign IP Addresses on a Control NetworkBuild an addressing plan before the first device is configured: one subnet per zone and site, a fixed block layout so an address says what the device is, static addresses on everything that controls a process, and a schedule that is kept current.
- Time SynchronizationWhy every clock in a control system must agree and what happens when they do not: out-of-order events, historian gaps at daylight saving, authentication failures, wrong totals. The time source and hierarchy, how each device is synchronized, and the checks.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.