The short answer
Patch Management
Patch management for a control system is a monthly cycle: the operating system and application updates are released, the SCADA vendor publishes which of them are qualified for its product, the qualified updates are applied to a test system with the application exercised, and then to production in a maintenance window, one server of a redundant pair at a time, with a backup or snapshot before each and a rollback plan. Priority runs from the most exposed machines inward: anything in the DMZ or reachable remotely first, then servers, then clients, with controllers and network devices on their own firmware cycle. Actively exploited vulnerabilities get an emergency path with the same test and backup steps compressed. Software that cannot be patched is isolated, allowlisted, and scheduled for replacement, and a record shows the patch state of every machine.
Key points
- Monthly cycle: release, vendor qualification, test system, maintenance window, production one machine at a time.
- The vendor qualification list is the gate; an unqualified patch on a SCADA server is a gamble with the plant.
- Edge first: DMZ and remote-reachable machines, then servers, then clients. Controllers have their own firmware process.
- Backup or snapshot before every patch, rollback plan written, application tested after.
- Exploited vulnerabilities get an emergency path, compressed but not skipped.
- Unpatchable systems are isolated, allowlisted, and on a replacement plan, not ignored.
Why control systems lag
An office computer installs updates overnight and reboots; if something breaks, the user calls the helpdesk. A SCADA server that installs an update and reboots at 2 a.m. drops every operator display; if the update breaks the application, the plant runs blind until someone rebuilds it. Vendors of SCADA software respond by testing each operating system update against their product and publishing the list of qualified updates, usually within weeks of release. Utilities respond by turning off automatic updates and running a deliberate cycle. The lag is a few weeks, which is acceptable; the lag becoming years, which is common, is not.
The cycle
- 1
Collect
Operating system updates, application updates, and third-party software updates released this month, and the vendor qualification notices for each product in the system.
- 2
Assess
Which apply to which machines, which are qualified, and which address vulnerabilities that are exposed on this system. Anything actively exploited and exposed goes to the emergency path.
- 3
Test
Apply the qualified set to the test system, a virtual copy of a server and a client, and exercise the application: polling, alarms, history, displays, reports, failover, remote access. A day of soak.
- 4
Schedule
A maintenance window agreed with operations, with the redundant pair patched one at a time and the clients patched while a second client remains available.
- 5
Back up
Snapshot or image of each machine before it is patched, and confirmation that the backup can be restored.
- 6
Apply
Standby server first; fail over; confirm; then the former active. Clients one at a time. Reboot as required, in the window.
- 7
Verify
The application check after each machine, and a look at the logs the next day for errors that appeared.
- 8
Record
Machine, updates applied, date, who, result, and any exception with its compensating control and its review date.
Priority
| Tier | Machines | Why first | Cadence |
|---|---|---|---|
| 1 | DMZ servers, remote access gateways, jump hosts, anything reachable from outside the control zone | They face the attackers | Within days of qualification; emergency within hours |
| 2 | SCADA servers, historian, database, domain controllers for the control system | They run the plant and hold the records | Monthly, in the window |
| 3 | Operator consoles, engineering workstations | Many hands touch them; they reach the servers | Monthly, rolling |
| 4 | Network equipment firmware: switches, firewalls, radios | Exposure varies; firewalls are tier 1 when internet-facing | Quarterly or on advisory |
| 5 | Controllers, remote units, drives, instruments | Firmware updates change behavior; tested and scheduled separately | On advisory and at planned outages; see firmware management |
The emergency path
When a vulnerability is being exploited in the wild and the system is exposed to it, waiting for the monthly window is the wrong risk. The emergency path runs the same steps in a day: the vendor qualification checked or the vendor asked, a quick test on the test system, a backup, and the patch applied to the exposed machines first. Where the patch is not yet qualified, the compensating control is applied immediately instead: the vulnerable service disabled, a firewall rule added, the machine isolated, and the patch follows when it is qualified. The decision and its reasoning are recorded either way.
What cannot be patched
Every utility has some: an HMI on an operating system out of support, a historian version the vendor no longer updates, a device whose firmware was last released a decade ago. They are not ignored; they are managed. Isolate the machine in its own segment with firewall rules that allow only what it must do; enforce application allowlisting so nothing new runs; remove it from any remote access path; monitor it; and put it on a replacement plan with a date. The record lists each one with its compensating controls, and the review each year asks whether the date has arrived.
Antivirus definitions
Antivirus signature updates are the one thing that does update daily, and they arrive through an internal update server in the DMZ that pulls from the vendor and serves the control network, so that no control machine talks to the internet. The same DMZ server distributes operating system updates once they are approved, which gives the cycle a delivery mechanism that does not involve a USB drive.
Frequently asked questions
- The vendor qualification takes weeks. Are we exposed in the meantime?
- The exposure depends on whether the vulnerability is reachable on the system. Most operating system vulnerabilities require network access to a service that a hardened, segmented server does not expose, and a few weeks of lag is acceptable. For the ones that are exposed and exploited, the emergency path and the compensating controls exist.
- Can we skip a month?
- A month skipped is two months of updates next time, tested and applied together, which is harder to troubleshoot and longer in the window. Skip only for a reason, and record it. A cycle that skips often is a cycle that has stopped.
- What is a test system for a small utility?
- A virtual machine with a copy of the SCADA server, and one with a client, on a spare host or even on the production hypervisor with restricted networking. The vendor license usually allows a test copy. Without one, the standby server of a redundant pair is the test system, which is acceptable with a snapshot; a single production server with no test system is patched with a backup and a prayer.
- Who approves the patch?
- The control system owner, on the recommendation of whoever runs the cycle, in the change management process. The approval is a record that says what was applied and why, which is what the auditor and the incident investigator will ask for.
Related topics
- Risk-Based PatchingDeciding what to patch first when everything cannot be patched at once: priority from exposure, exploitation, and consequence rather than scores, tiers with timelines, different rules for servers, network gear, and controllers, and the written accept decision.
- Server HardeningReducing what an attacker can do with a SCADA server: a baseline from the vendor guide and a benchmark, unused services removed, the host firewall on, application allowlisting, controlled media, separate administrator accounts, and logs sent off the box.
- Firmware ManagementUpdating the firmware of controllers, modules, drives, HMIs, switches, and radios without breaking the plant: why firmware lags and why it still matters, tracking versions, deciding what to update and when, and the update as a planned outage with a rollback.
- Compensating ControlsWhat to do about a vulnerability that cannot be patched now or ever: restricting who can reach it, disabling the service, blocking dangerous protocol functions, allowlisting, monitoring, physical and procedural controls, matched to the attack path.
- SCADA BackupsWhat a SCADA backup must contain to bring a system back: the project export, an image of each server, the databases, historian archives, licenses, certificates, scripts, and network device configurations, with frequency, retention, and the restore procedure.
- Change Management for OTA change process sized for a utility control system: what counts as a change, the request with risk and rollback, owner approval, the window, backup before and verification after, the baseline update, the emergency path, and the record.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.