The short answer
Backup Testing
A backup that has never been restored is an assumption, and the assumptions that fail are always the same: the image will not boot on different hardware, the license refuses the new machine, the archives were not in the backup, the project export is a version the installed software cannot read, the encryption key is with the person who left, the media is corrupt, and nobody knows the procedure. Backup testing restores each kind of backup on a schedule: a file from the last backup every month, a full server to an isolated virtual machine with the application exercised every quarter or at least every year, a controller program to a spare or test controller with a compare against the running one, a historian archive from the oldest year opened in a trend, and a network device configuration loaded onto a spare. Each test is timed, its findings are fixed, and the record shows that the utility can recover and how long it takes.
Key points
- The backup is proven by the restore; nothing else counts.
- Test every kind: file, server image, project export, controller program, historian archive, device configuration.
- Restore to an isolated machine and exercise the application; a server that boots is not a server that works.
- Time it; the result is the recovery time the utility can actually promise.
- Every test produces findings; fixing them is the point of testing.
- Rotate who runs the test so that more than one person can recover the system.
How untested backups fail
- The image was taken from physical hardware and will not boot on the replacement, or the virtual machine version is newer than the host that must run it.
- The application starts and demands a license that was tied to the old machine identity; the vendor reactivation takes days.
- The historian restores with no history because the archive files were never in the backup job.
- The project export is from a version two releases newer than the installer that is available.
- The backup is encrypted and the key was in a password manager belonging to an administrator who left.
- The external drive that held the offline copy has not been plugged in for a year and no longer reads.
- The procedure was in the head of one person, and the person who is doing the restore at 3 a.m. is not that person.
- The database backup is a week older than the project export, and the two do not agree.
The tests
| Test | What it proves | Frequency | How |
|---|---|---|---|
| File spot restore | The backup job produces readable data and the media works | Monthly | Restore a project export, a configuration file, and a database backup from last week to a scratch location and open them |
| Full server restore | The image or the reinstall-and-import path produces a working server | Quarterly for critical servers; at least annually for all | Restore to an isolated virtual machine with no connection to production; start the application; exercise polling against a simulator, alarms, displays, history, reports |
| Project import on a fresh install | The clean-rebuild path works, for the day the image cannot be trusted | Annually | Install the platform from the retained installers on a clean machine, import the project export, apply the license |
| Controller program restore | The stored program matches the running one and can be loaded | Annually per controller type; after every change on critical controllers | Compare the backup with the running controller; download to a spare or test controller and verify it runs |
| Historian archive read | Old archives are present, registered, and readable | Annually | Open a trend from the oldest year on the restored historian, or register an old archive on a test historian |
| Network device restore | The configuration export loads and produces the same behavior | Annually | Load onto a spare switch or firewall on the bench and compare |
| Offline copy restore | The disconnected copy is complete and readable | Semi-annually | Perform one of the above from the offline copy rather than the online one |
| Credential recovery | The break-glass envelope and the vault export work | Annually | Open the vault export on an isolated machine with the sealed credentials; reseal with new ones |
Running a server restore test
- 1
Isolate
A virtual machine on a host or a network segment with no path to production, so that the restored server cannot poll real controllers or serve real clients. A simulator or a test controller stands in for the field.
- 2
Restore
From the backup of record, following the written procedure exactly, by someone other than the person who wrote it. Note every step where the procedure was wrong or unclear.
- 3
License
Apply the license as the procedure says; if it requires the vendor, do that call, and record how long it took.
- 4
Exercise
Start the application. Poll the test controller. Raise an alarm and acknowledge it. Open the displays on a test client. Trend history including a period from before the backup. Run a report. Fail over if the pair is being tested.
- 5
Time
From the start of the restore to a working application. That is the recovery time.
- 6
Record and fix
The findings, the corrections to the procedure and the backup job, and the date of the next test. Findings are assigned to someone with a date.
The record
Each test produces a one-page record: what was restored, from which backup, onto what, by whom, how long it took, what worked, what did not, and what was changed as a result. The records over a few years show whether recovery time is improving, and they are what an auditor, an insurer, or a regulator asks for when they ask whether the utility can recover. A test that found nothing wrong is either a mature system or a test that did not look hard enough; the second is more common.
Frequently asked questions
- We do not have a spare server to restore onto. How do we test?
- A virtual machine on any host with enough memory, including a workstation with hypervisor software, is enough for a restore test of a small system; the test is functional, not performance. The vendor license normally allows a test copy. If there is truly nowhere to restore to, the utility has no recovery plan, and the test has already found its first finding.
- How long should a restore take?
- An image restore of a virtual machine, minutes to an hour; a reinstall and project import, several hours; a full rebuild of a multi-server system, a day or two. The measured time from the test is the number that matters, and it is what the recovery time objective is set against, not the other way round.
- Is restoring a controller program to the running controller a test?
- No, it is a risk. Compare the backup with the running program using the vendor tools, which proves the backup matches; download to a spare or a test controller to prove it loads. Downloading a backup to a running controller is done only when the controller has failed.
- What if the test fails badly?
- Then the test has done its job, on a day when nothing was actually lost. Fix the backup job, the procedure, or the missing piece, and test again within a month. A failed test is a success of the testing program; the failure that matters is the one that happens during a real recovery.
Related topics
- SCADA BackupsWhat a SCADA backup must contain to bring a system back: the project export, an image of each server, the databases, historian archives, licenses, certificates, scripts, and network device configurations, with frequency, retention, and the restore procedure.
- Offline CopiesThe backup copy that survives ransomware: why anything reachable over the network will be encrypted too, what offline and immutable mean, options from rotated drives to immutable storage, what goes in the offline set, and the restore test that uses it.
- PLC Program BackupsWhat a complete controller backup contains, how often to take one, where to keep it, why the upload from the running controller is not the master, and the verification that turns a folder of files into a recovery plan.
- Recovery and RestorationBringing a control system back after a compromise without reintroducing it: controllers verified against baselines first, servers rebuilt from clean media and project exports, every credential changed, hardening, staged reconnection, and end-to-end tests.
- Server FailureHow SCADA servers fail and what to do when one does: failure modes from a full disk to a dead hypervisor host, what the plant experiences with and without redundancy, the recovery sequence from restore to alarm verification, and the monitoring that warns.
- Credential StorageWhere the passwords and keys of a control system are kept: a vault with individual access and a log, break-glass copies in a safe, platform credential stores for service accounts, and the places they must not be, from spreadsheets to project files.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.