Recovery and Restoration

When to start

Recovery begins when the investigation has established how the intruder got in and what they touched, or has at least established the boundaries of what cannot be trusted. Starting earlier rebuilds systems that are reinfected from a machine nobody looked at. While the investigation runs, the plant operates by the manual procedures, and the recovery team prepares: clean installation media verified, the offline backups retrieved and checked, spare hardware or virtual machines staged, credentials generated, and the sequence written down. The pressure to reconnect is real and it is resisted until the plan says a part is clean.

The sequence

1. 1

   Controllers first

   For each controller and remote unit: compare the running program against the baseline with the vendor tools. A match, and a review of the data and setpoints, means it can be trusted. A difference means the program is reloaded from the offline copy, the firmware is verified or reloaded, the controller credentials are changed, and the process is watched in hand through a cycle before the controller is returned to automatic.

2. 2

   Servers from clean media

   Each server rebuilt on a fresh virtual machine or wiped hardware from verified installation media, the operating system hardened to the baseline and patched, the application installed at a supported version, and the project imported from an export that predates the compromise or has been reviewed. Images taken during the intrusion are kept for evidence and not restored.

3. 3

   Data

   The historian archives restored from backups, with the period of the intrusion treated as suspect and marked; the alarm journal and audit database restored to the last clean backup and the incident period reconstructed from the collector logs.

4. 4

   Credentials

   Every account password, every service account, every device password, every key and certificate, and the vault itself rebuilt with new master credentials. Any account the intruder could have created is found and removed.

5. 5

   Network

   Firewall and switch configurations restored from baseline and reviewed rule by rule; the rule or the path that was used is closed; monitoring in place at the boundaries before anything is reconnected.

6. 6

   Reconnect in stages

   Control zone first, with the rebuilt servers polling the verified controllers and the operators watching; then the supervisory zone; then the DMZ with the historian replica; then remote access, last, with multi-factor authentication confirmed. At each stage, a pause and a look at the monitoring for anything unexpected.

7. 7

   Test

   Loop checks on critical signals, alarm tests end to end including notification, failover of the redundant pair, and a walk of the displays against the plant.

8. 8

   Return to automatic

   Process by process, with the manual log closed for each, the setpoints checked, and an operator watching the first cycles.

9. 9

   Close

   The incident declared over by the incident lead with management, the reports filed, the evidence retained, and the review scheduled.

Rebuild, not restore

The instinct after an intrusion is to restore the server image from last night and get the screens back. The image from last night contains whatever the intruder installed last week, and restoring it returns the system to them. The clean path is longer: install from media whose hashes have been verified, apply the hardening baseline, patch, and import the project from an export that has been reviewed, or from before the earliest sign of intrusion. The project export is data and configuration rather than executable code, which is why it can be trusted where the image cannot, and it is why keeping project exports as well as images matters. The rebuild also produces a system that is hardened and patched, which the original may not have been.

Testing before trust

• Every controller: program compare, firmware version, mode, and a watched cycle of the process.
• Critical loops: a loop check from the instrument to the display, and a command from the display to the equipment.
• Alarms: the highest priority alarms forced and followed to the display, the journal, and the phone.
• Communications: every remote site polling, with quality good and timestamps correct.
• Redundancy: a failover of the server pair and a check of alarm states afterward.
• Accounts: every login tested for its role; no account that was not created by the rebuild.
• Backups: the first backup of the rebuilt system taken, and an offline copy made, before the recovery is declared complete.

After

The review is held within weeks, with everyone who was involved, and it produces changes: to the network design that allowed the entry or the spread, to the procedures that were unclear, to the backups that were missing something, to the monitoring that did not see it, to the plan that did not name a person. The incident report goes to management and, as required, to the regulators and the sector organizations, and the lessons are shared where the utility can share them. The recovery time, from isolation to the last process in automatic, is measured and becomes the new planning number.