Symptom
The entire network, or an entire segment, becomes slow or unreachable within seconds; every switch link light is on or flickering constantly; switch processors run at maximum; controllers and workstations lose communication together; the event began when a cable was connected, a switch was added, or a redundancy link was disturbed.
The short answer
Broadcast Storm
A broadcast storm is a network drowning in frames that every switch forwards to every port: a loop between switches with no spanning tree or ring protocol to block it makes each broadcast circulate forever and multiply, and within seconds every link is saturated, every switch processor is at its limit, and every device is either unreachable or too busy dropping frames to work. A single misbehaving device can do the same at a smaller scale by transmitting broadcasts or multicasts continuously. The signs are unmistakable: all link lights solid, switch processor at maximum, the same broadcast frame captured thousands of times, and everything failing at once. The immediate cure is to break the loop by disconnecting links until the network recovers; the lasting cure is loop protection, storm control, and a physical topology that nobody can accidentally close.
Key points
- Everything failing at once, with every link light solid, is a storm until proven otherwise.
- A loop with no protection is the usual cause; the second cable someone plugged in is the usual loop.
- Break it first: pull uplinks one at a time until the network recovers, then find the loop at leisure.
- Spanning tree or a ring protocol, storm control, and disabled unused ports make a storm nearly impossible.
- A single device flooding broadcasts shows up in a capture and on one access port counter.
Possible causes and what to check
| Possible cause | What to check |
|---|---|
| Loop from an extra cable | A patch cord between two ports of one switch, or between two switches already connected; often at a patch panel or in a temporary cabinet. Look for the newest cable. |
| Loop through an unmanaged switch or a device with two ports | A small unmanaged switch under a desk connected twice, or a device with a built-in two-port switch cabled to two access ports. Trace the device. |
| Spanning tree disabled or misconfigured | Redundant links between switches with the protection turned off or with mismatched protocol versions. Read the switch configuration. |
| Ring protocol failure | A ring whose manager failed or lost its protocol frames, leaving the ring closed. The ring status on the manager. |
| Wireless bridge closing a loop | A bridge that links two segments already connected by wire. Check the bridge topology. |
| Device with a failed interface flooding | One access port with an extreme frame rate; a failing network interface or a device in a fault. Disable the port. |
| Software or misconfiguration flooding | A discovery tool, a misconfigured video multicast, or a poll to a broadcast address at high rate. Capture and find the source address. |
| Address resolution storm | A device or scanner probing every address at high rate; the switch counters and a capture show the source. |
| Switch hardware fault | A switch that repeats frames or has a failed backplane; storms that persist with every external link removed. Replace. |
Recognising a storm
| Sign | Storm | Something else |
|---|---|---|
| Scope | Everything on the segment at once | One device, one link, one switch |
| Link lights | All solid or flickering constantly | Normal activity on most ports |
| Switch processor | At maximum | Normal |
| Onset | Seconds, often after a cable or a switch change | Gradual, or after a device change |
| Capture | The same broadcast frame thousands of times per second | Normal broadcast rate of a few per second |
| Recovery | Instant when a link is pulled | Unchanged by pulling links |
Breaking it
- 1
Protect the process first
Controllers keep running on their own logic through a storm, but supervisory setpoints, interlocks across the network, and operator control are gone. Confirm the plant is safe in local control before doing anything else.
- 2
Pull the newest cable
If a cable was just connected or a switch just added, remove it. Most storms end here.
- 3
Pull uplinks one at a time
At the core switch, disconnect uplinks to distribution or access switches one at a time, waiting a few seconds each; when the storm stops, the loop is downstream of the last link pulled. Reconnect the others.
- 4
Divide again
On the switch that was isolated, repeat with its ports until the loop is found: two ports whose removal ends it, or one access port carrying the flood.
- 5
Find the physical cause
Follow the cable from the port; the extra patch cord, the unmanaged switch cabled twice, the device with two ports, or the flooding device.
- 6
Restore in order
Reconnect links from the core outward, watching the processor load and the broadcast counters at each step.
Finding a flooding device
When the flood comes from one device rather than a loop, the switch counters show one access port with a frame rate far above the others, and a capture on any port shows the source hardware address repeated. Disable that port, confirm the network recovers, and go to the device. A failing network interface, a device in a firmware fault, a discovery tool left running, a video system multicasting without a group management protocol, and a scanner probing every address have all done this.
Making it impossible
- Spanning tree, or the ring protocol the switches support, enabled on every switch, with a deliberate root and a documented topology. Redundant links exist to be blocked until needed.
- Loop protection on access ports: a port that receives its own protocol frames shuts itself down, which catches the patch cord under the desk.
- Storm control on every port with a threshold above normal broadcast levels; a port that exceeds it is rate limited or shut, and an alarm is raised.
- Unused ports disabled, so that a stray cable does nothing.
- No unmanaged switches on the control network; a device that needs two ports gets a managed switch with protection.
- A network drawing that shows every redundant link and every ring, kept current, so that the person with the cable knows what they are closing.
- Broadcast rate monitored and alarmed by the network monitor, so that a slow flood is seen before it becomes a storm.
Frequently asked questions
- The controllers kept running through the storm. Was it a problem?
- Yes. Everything that depends on the network stopped: operator control, supervisory setpoints, interlocks between controllers, alarms, and the historian. A controller that runs blind on its last setpoint is not the same as a plant under control.
- We have spanning tree. How did we get a loop?
- A ring protocol and spanning tree running on different switches without agreement, a switch with the protocol disabled on a port, an unmanaged switch that does not participate, or a media converter that drops the protocol frames. Read the configuration of every switch in the loop and enable protection everywhere.
- Can a storm damage equipment?
- Not the hardware, but it can put a controller into a fault state when a network interface is overwhelmed, drop a redundant controller pair out of synchronization, and fill a switch log with events. The consequences are process consequences.
- Why did it start when the electrician plugged in the camera?
- Because the camera, or the small switch it came with, had two ports and both were cabled, or because the cable from the camera location was already connected at the other end. Loop protection on the access port would have shut it down in a second. Enable it.
Related topics
- Intermittent Packet LossPolls that occasionally fail, tags that flicker bad, a ping that drops one in fifty: a bad cable or port, a duplex mismatch, a saturated link, a buffer overrun, a marginal radio or fiber hop, a slow device, or a loop. Finding which hop loses packets and why.
- Switch Port Errors IncrementingWhat each counter on a managed switch port means and which fault makes it grow: check errors from a damaged cable or noise, late collisions from a duplex mismatch, runts from a bad transceiver, discards from congestion, link flaps from a marginal connection.
- Ethernet Device Drops OfflineAn industrial Ethernet device that disappears and comes back. Duplex mismatch, cabling, spanning tree, and the switch counters that identify it in minutes.
- SCADA RedundancyWhat redundant SCADA servers protect against and what they do not, how failover works for tags, alarms, history, and clients, the network and controller layers beneath it, and why an untested failover is not redundancy.
- Network DrawingsThe two drawings every control network needs: a logical drawing of zones, subnets, VLANs, firewalls, and conduits, and a physical drawing of switches, ports, cables, fiber, radios, and sites. What each shows and why they are sensitive.
- VLAN SegmentationUsing VLANs to build the zones of a control network on shared switches: what a VLAN separates and what it does not, the zone-to-VLAN mapping, trunks and the native VLAN, routing only through a firewall, and where separate hardware begins.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.