The short answer
Server Failure
SCADA servers fail through hardware faults, operating system problems, application service crashes, expired licenses, full disks, memory exhaustion, network interface or switch faults, and failures of the virtualization host or storage under them, and the most common of these announce themselves for days in monitoring that nobody watched. With redundancy the standby takes over and the failure is a repair; without it the plant runs on local control while operators lose visibility, alarms, and notification. Recovery restores the server from an image or a backup, reactivates licenses, restarts services in order, reconnects clients, backfills the historian from remote buffers, and verifies alarm states and acknowledgments. A written recovery procedure with a target recovery time, a tested restore, and monitoring of disk, memory, services, and backups are what turn a failure into an inconvenience.
Key points
- Full disks, expired licenses, and unpatched crashes cause more outages than dead hardware, and all three give warning.
- Without redundancy the plant runs blind: local control continues, alarms and notification stop. Know which sites cannot tolerate that.
- Recovery order: restore, license, services, clients, historian backfill, alarm state verification.
- A restore that has never been tested is a hope, not a plan.
- Monitor disk, memory, services, backup completion, license expiry, and time; alarm them like process alarms.
- Write the recovery procedure with the recovery time objective, and keep it where it can be read with the server down.
How servers fail
| Failure | Symptom | Warning available | Prevention |
|---|---|---|---|
| Disk full | Services stop, historian stops, database errors, logins fail | Weeks of disk usage trend | Disk alarms; log rotation; retention limits; historian sizing |
| License expired or lost | Runtime stops or goes to demo mode on a restart or at midnight | Expiry date in the license record | License calendar; keys and activation files in the backup |
| Application service crash | One function stops: polling, alarms, history, clients | Event logs, memory growth over days | Patching; service watchdogs; scheduled restarts where the vendor advises |
| Operating system fault or forced update reboot | Server reboots or hangs | Update settings | Managed patching; updates never automatic on a control server |
| Hardware: disk, power supply, memory | Crashes, errors in the hardware log, a dead server | Hardware monitoring, RAID alerts | Redundant disks and supplies; hardware alerts to the alarm system; lifecycle replacement |
| Network interface or switch port | Server up, nothing can reach it | Link errors | Dual interfaces; switch monitoring |
| Hypervisor host or storage | Every virtual machine on it stops | Host alerts, storage alerts | Two hosts with the redundant pair split; storage redundancy |
| Database full or corrupt | Alarm journal and audit trail stop; the application may stop | Database size trend | Retention and purge jobs; database backups |
| Malware or intrusion | Anything, including a ransom note | Security monitoring | Segmentation, hardening, backups off the network |
What the plant experiences
With a redundant pair, the standby becomes active, clients reconnect, and operators see a brief interruption; the failed server is repaired and resynchronized. The event is still investigated, because a pair with one dead member has no redundancy. With a single server, the controllers continue to run the process on their own logic, which is why local control is designed to need nothing from the server. What stops is visibility, alarming, notification, and history. A lift station that overflows during the outage does so silently; a chlorine feed that fails is not called out. The utility should know, in advance, which sites and processes cannot go unwatched for the length of a server recovery, and what the manual procedure is: an operator visiting sites, a local alarm dialer independent of the server, a second small system for the critical points.
Recovery
- 1
Stabilize the plant
Confirm local control is working at critical sites; dispatch someone to watch what cannot be seen. This comes before any server work.
- 2
Diagnose
Hardware alive? Operating system booting? Services running? Disk full? License valid? Network reachable? The failure table says what to look at; the event logs say what happened.
- 3
Repair or restore
A full disk is cleared and its cause fixed. A crashed service is restarted. A dead server is rebuilt from the image backup onto the spare hardware or a new virtual machine, or the application is reinstalled and the project restored from the application backup.
- 4
License
Reactivate or reinstall the license, which for many products is tied to the hardware or the virtual machine identity and needs the vendor. Keep the activation records with the backup.
- 5
Services in order
Database, then tag server and drivers, then alarm and historian, then client services. Confirm each is running and connected before the next.
- 6
Clients
Reconnect the operator stations and confirm the displays are live, with communication status good.
- 7
Historian backfill
Trigger or verify the backfill from remote controllers with buffers; note the gap for the record.
- 8
Alarms
Review the active alarm list against the plant: alarms that should be active but are not, acknowledgments lost, shelved and out-of-service states restored.
- 9
Record
What failed, when, how long, what was done, what was lost, and what will prevent it. The recovery procedure is updated from this.
The monitoring that gives warning
- Disk free space on every volume, alarmed at a threshold weeks before full.
- Memory and CPU trends; a service whose memory grows daily is a leak with a date.
- Service status for every SCADA service, alarmed on stop.
- Backup job completion and age of the last good backup.
- License expiry date, with a reminder a month out.
- Hardware and hypervisor alerts routed to the alarm system.
- Time synchronization status.
- Poll success rates and historian queue depth, which fall before a driver or collector dies.
Frequently asked questions
- The server is fine but nobody can log in. Is that a server failure?
- From the plant point of view, yes: visibility is lost. Common causes are a full disk, an authentication service or domain controller unreachable, an expired certificate, or a client license count exceeded. Check the disk first, then authentication; the server event log usually names it.
- How long should recovery take?
- As long as the recovery time objective the utility has decided it can tolerate, which depends on what runs unwatched. A few hours is achievable with a tested image restore and a spare virtual machine; days is what happens without them. Decide the objective, then buy and practice the recovery that meets it.
- What is the minimum for a small utility with one server?
- An image backup on a schedule kept off the server, the application project exported and kept with the license records, a spare machine or a hypervisor that can host a restore, a documented procedure, one practiced restore a year, and disk and service monitoring. That set costs little and covers most outages.
- Should the alarm dialer depend on the SCADA server?
- Not entirely. A dialer or notification system that takes its alarms only from the server stops when the server does. A hardware dialer at critical sites, wired to local alarm contacts, or a second small notification path that watches the server itself, is the layer that calls someone when the SCADA cannot.
Related topics
- SCADA ServersWhat the servers in a SCADA system do and what a utility should expect of them: the I/O, alarm, historian, and client roles, physical versus virtual, redundancy, sizing, placement, backup, patching, licensing, and monitoring.
- SCADA RedundancyWhat redundant SCADA servers protect against and what they do not, how failover works for tags, alarms, history, and clients, the network and controller layers beneath it, and why an untested failover is not redundancy.
- What to Back Up in a Control SystemWhat a utility must be able to restore after a failure or attack: controller programs and firmware, SCADA and historian, HMI panels, drive and instrument parameters, network configurations, licenses, installers, and documentation, and how often each changes.
- OT Incident Response PlanThe plan for the day the control system is not trusted: who decides, the triggers, the first hour, isolating without stopping treatment, running by hand, preserving evidence, who to notify and when, recovery from backups, and the exercise that makes it real.
- Historian ProblemsHow a historian goes wrong and how to find each fault: gaps, wrong values, detail never stored, disks that fill, slow queries, backfill that never arrives, and unreadable archives, with the collector, queue, configuration, and storage checks that locate it.
- Alarm Notification and CalloutGetting the alarm to the person on call when no one is watching the screen: notification paths, escalation, acknowledgment from the field, which alarms qualify, and the failure modes that leave a station in high level with nobody paged.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.