The short answer
Offline Copies
Ransomware encrypts every backup it can reach, which includes network shares, backup servers whose credentials are on the compromised machines, and cloud folders that synchronize. An offline copy is one that no process on the network can reach: a drive that is physically disconnected and locked in a safe, a tape that has been ejected, or storage configured so that written data cannot be altered or deleted for a set period. The offline set holds what a rebuild needs: controller programs, the SCADA project export, server images, historian exports, network device configurations, licenses, installers, the credential vault export, and the documentation. It is refreshed on a schedule that matches how much the utility can afford to lose, handled with labels, encryption, and a log, and it is the copy that the restore test uses at least once a year.
Key points
- If a process on the network can write to it, ransomware can encrypt it. Offline means unreachable.
- Rotated external drives in a safe are a legitimate offline copy for a small utility; immutable storage or tape scales up.
- The offline set is everything a clean rebuild needs, including licenses, installers, and the vault export.
- Refresh on a schedule set by what can be lost; weekly is common, daily for the project export after changes.
- Label, encrypt, log, and rotate the media; verify it reads.
- Restore from the offline copy at least once a year; it is the only copy that matters on the worst day.
Why the online backup is not enough
Modern ransomware is operated by people who spend days inside a network before they act. They find the backup server, the credentials that reach it, the network shares, and the cloud synchronization, and they delete or encrypt the backups first, then the servers. A utility whose backups were all online discovers on the same morning that the SCADA is encrypted and so is every copy. The offline copy is the one they could not find because it was not connected to anything. It is a simple idea and it is the single most valuable control against the most common attack on utilities.
What offline means
| Approach | How it is offline | Fits | Watch out for |
|---|---|---|---|
| Rotated external drives | Physically disconnected after the copy; two or more drives rotated; kept in a safe, one off site | Small utilities; simple and cheap | A drive left plugged in is online; drives fail; encrypt them; log the rotation |
| Tape | Ejected cartridges; a library with a write-once option | Larger systems with a lot of history | A tape drive to read them in ten years; store correctly |
| Immutable storage | Storage that refuses modification or deletion for a retention period, enforced by the storage system | Utilities with an IT function; cloud object storage with object lock | Immutability enforced by the storage, not by a permission that an administrator account can change; the administrator account will be compromised |
| Isolated backup server | A backup server on its own network, reachable only through a one-way path, with no domain trust | Medium systems | Isolation that is real; one-way means the server pulls or a diode pushes |
| Printed and physical | Setpoints, drawings, procedures, credentials in the safe | Every utility | It has to be current |
The offline set
- Controller programs for every controller and remote unit, in the vendor project format, verified against the running controller.
- The SCADA project export and the server images.
- Historian archive files or an export of the compliance data.
- Firewall, switch, router, and radio configurations.
- License files, activation records, and vendor account details.
- Installers for the SCADA version, the operating system, third-party software, and the controller programming tools.
- The credential vault export, encrypted, with its key in the break-glass envelope.
- The documentation: functional descriptions, drawings, the IP schedule, the restore procedures, the incident response plan, and the manual operation procedures, on paper as well.
Schedule and handling
The refresh interval is set by how much change the utility can afford to redo: a week of lost SCADA changes is recoverable, a year is not. Many utilities refresh the offline set weekly with an automated copy to a drive that is then disconnected, and add the project export after every change. Media is labeled with the date and the contents, encrypted with a key that is in the vault and the safe, logged when it goes into the safe and comes out, stored in a fire-rated safe with one copy off site, and replaced on a schedule before it fails. A drive that has not been read in a year is checked before it is trusted.
- 1
Decide the set and the interval
The list above, trimmed or extended for the system, and a refresh interval written in the backup policy.
- 2
Choose the method
Rotated drives, tape, immutable storage, or an isolated server, by the size of the system and the skills available.
- 3
Automate the copy
A job that assembles the set and writes it to the medium, monitored for completion, and a person who disconnects and stores the medium, with a checklist.
- 4
Encrypt and label
Encryption on the medium; the key in the vault and the safe; a label with date and contents.
- 5
Rotate and store
Two or more media in rotation; the safe on site and a second location off site; a log.
- 6
Test from it
At least once a year, a restore that uses the offline copy and nothing else.
Frequently asked questions
- Is cloud backup an offline copy?
- Only if the cloud copy is immutable with object lock or equivalent for a retention period, under an account separate from the ones on the network, with multi-factor authentication. A cloud folder that synchronizes from a server is online and will synchronize the encryption. Even with immutable cloud storage, keep a physical offline copy of the small critical set: controller programs, project export, licenses, and the vault.
- How much does an offline copy cost?
- For a small utility, two or three encrypted external drives, a fire-rated safe, and an hour a week. It is the cheapest control in the security program by a wide margin, and it is the one that has determined whether utilities recovered from ransomware in days or months.
- Who keeps the off-site copy?
- A second utility building, a bank safe deposit box, or a trusted arrangement with a neighboring utility, under the same encryption and log. Not the home of an employee, who may be unreachable or may leave.
- What about paper?
- The procedures for running the plant by hand, the drawings, the setpoints, and the sealed credentials are in the safe on paper, because on the day the offline copy is needed there may be no working computer to read a drive on. Paper is the last offline copy.
Related topics
- SCADA BackupsWhat a SCADA backup must contain to bring a system back: the project export, an image of each server, the databases, historian archives, licenses, certificates, scripts, and network device configurations, with frequency, retention, and the restore procedure.
- Backup TestingWhy backups that have never been restored fail when needed, and how to find out: spot restores, full restores of a server to an isolated machine, controller program restores to a spare, archive reads, timing each one, and the findings and record that follow.
- PLC Program BackupsWhat a complete controller backup contains, how often to take one, where to keep it, why the upload from the running controller is not the master, and the verification that turns a folder of files into a recovery plan.
- Credential StorageWhere the passwords and keys of a control system are kept: a vault with individual access and a log, break-glass copies in a safe, platform credential stores for service accounts, and the places they must not be, from spreadsheets to project files.
- Recovery and RestorationBringing a control system back after a compromise without reintroducing it: controllers verified against baselines first, servers rebuilt from clean media and project exports, every credential changed, hardening, staged reconnection, and end-to-end tests.
- Water and Wastewater Utility Threat LandscapeWhat has actually happened at water utilities, what those incidents had in common, and the small number of controls that would have prevented most of them.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.