One Flat Network Is One Big Fault Domain

Somebody in the administration building plugged both ends of a patch cable into the same small desktop switch. It took about a second. The unmanaged switch forwarded every broadcast frame back to itself forever, the storm spread through the building switch to the plant switch, and every PLC on the site lost its connection to SCADA as its network port was buried under traffic. The operators saw every screen go stale at once. The technician spent two hours at the plant looking for a failed switch before anyone thought to ask what was happening in the office. The plant network and the office network were the same network, so the office fault was the plant fault.

What flat means

A flat network is one where every device is in the same broadcast domain: one subnet, switches connected to switches, no router or VLAN boundary between the operator's laptop and the lift station controller. It is the network every plant starts with, because it is what happens when switches are added as devices arrive. It works, and it works right up to the day something misbehaves, because in a flat network there is no such thing as a local problem. A broadcast frame reaches every port. A storm reaches every port. A device with a duplicate address confuses every device that talks to the real one. A laptop with malware can reach every controller by address. The size of the fault is the size of the network.

Segments for a small plant

Segmentation does not mean a dozen subnets and a network engineer on staff. A small plant is well served by a handful of VLANs, chosen by function and by what should be allowed to talk to what. A pattern that works: one segment for controllers and I/O, one for the SCADA servers and historian, one for operator workstations and the engineering laptop, one for remote sites arriving over radio or cellular, and one for everything that does not belong on the others, such as cameras, HVAC, and the printer. The office network is not a segment of the plant network. It is a different network, joined to the plant only through a firewall that allows what the plant chooses.

Controls

PLCs, remote I/O, drives with Ethernet, and instrumentation with Ethernet. Nothing in this segment initiates a connection to the internet, ever.

SCADA servers

The SCADA and historian servers, which talk to the controls segment by a short list of protocols and ports and to the clients by another.

Clients and engineering

Operator workstations, the programming laptop, and the maintenance port on the wall. This segment can reach the servers, and the engineering laptop can reach the controllers through a rule that exists for it.

Remote sites

Whatever arrives over radio, cellular, or leased line. Treated as less trusted than the plant, because the far end of a radio link is a cabinet in a field.

Other

Cameras, HVAC, access control, the printer. Things that were never designed for a control network and should not share one.

The hardware that makes it work

VLANs need managed switches, and a managed switch is also the thing that stops the loop. Rapid spanning tree blocks the redundant path that a looped cable creates; storm control caps broadcast traffic on a port before it can bury the segment; port security can refuse an unknown device on the maintenance port. IGMP snooping keeps the multicast traffic that some I/O protocols use inside the ports that asked for it, instead of flooding every port on the switch. Routing between segments needs either a layer 3 switch or a firewall, and for a plant the firewall is the better choice, because the point of the boundary is to decide what crosses it. Unmanaged switches have one remaining place: none. The small desktop switch under the operator's desk is where the loop happened.

An address plan

Each segment gets its own subnet, and the subnets should be chosen so that an address tells a person what it is. A third octet per segment, a consistent range for controllers within it, static addresses for everything that controls anything, and a written plan that lives with the network drawing. Devices that arrived with a default address get changed on the bench, not on the plant floor. The plan is what makes a duplicate address a rarity rather than a monthly event, and it is what the firewall rules are written against.

Getting there without a shutdown

1. 1

   Draw what exists

   Every switch, every uplink, every device with an address. The drawing usually reveals a few switches nobody knew about.

2. 2

   Put the firewall between the office and the plant first

   This is the biggest single gain, and it can be done with the plant still flat: one new subnet for the plant side, one rule set, and the office faults stop arriving.

3. 3

   Replace the core switch with a managed one

   Configure the VLANs on it before any device moves. Enable rapid spanning tree and storm control from the start.

4. 4

   Move one segment at a time

   Start with the servers, then the clients, then the controllers site by site, updating addresses and the firewall rules as each group moves. Each move is a short outage for the devices moving and none for the rest.

5. 5

   Retire the unmanaged switches

   Every one of them, including the ones under desks. Replace with managed switches or with ports on the nearest managed one.

6. 6

   Write the rules down

   The address plan, the VLAN table, the firewall rules, and the reason for each. The next technician should be able to add a device without guessing.

What it costs and what it saves

Managed industrial switches cost more than unmanaged ones, and a firewall is a line item that did not exist before. Against that, count the plant-wide outage that a looped cable causes, the hours spent chasing a fault that was in another building, and the fact that every cybersecurity guidance document for water utilities starts with segmentation because it is the control that makes the others possible. The plant in the story spent a modest sum on a firewall and three managed switches. The next looped cable, and there was one, took down a printer.