The short answer
The Backup That Would Not Restore
A PLC backup that works is the complete set of things a replacement controller needs to run the process the same way: the program with its comments, the firmware version it was written for, the hardware and I/O configuration, the communication settings, the retentive values and setpoints, and the software that opens all of it. A program file by itself restores the logic and nothing else. The only proof that a backup restores is restoring it, to a spare, and letting it run the process before the day it is needed.
Key points
- The program file is one part of a backup; firmware, configuration, setpoints, and the software that opens them are the rest.
- A spare controller with different firmware will refuse the program or run it differently; record the version with the file.
- Setpoints and retentive values live in the controller, not the program, and are lost unless they are saved separately.
- Keep an offline copy on media that ransomware cannot reach, and keep the programming software with it.
- A backup is proven only by a restore test onto the spare; schedule one and log the result.
The processor faulted at four on a Friday afternoon and would not clear. The utility had done the right things: a spare of the same model was on the shelf, and a copy of the program was on the network drive, saved by the integrator at startup six years earlier. The operator swapped the processor, the technician opened the file on the laptop, and the download failed. The spare had shipped with a newer firmware than the program was written for, and the software on the laptop was two versions behind the file. By the time the right firmware, the right software, and the right file were in the same place, it was Sunday, and the plant had run on floats and a man in a truck for two nights.
Nothing about that weekend was unusual. It is the normal shape of a PLC failure at a small plant, and it happens because the word backup is used for a file when it should be used for a kit. The program file is the part everyone remembers to save. It is also the part that is least likely to be the problem.
What was in the file and what was not
| Item | In the backup | What happened without it |
|---|---|---|
| Program logic | Yes | The logic was fine once it finally loaded |
| Program comments and tag descriptions | Partly | The startup copy had them; the copy the technician found first did not |
| Firmware version the program targets | No | The spare ran newer firmware and refused the download |
| Programming software of the right version | No | The laptop could not open the file until the vendor portal was visited |
| Hardware and I/O configuration | Yes, inside the program | Fine, but only because this platform stores it there |
| Communication settings and IP addresses | No | The spare came up at a default address SCADA could not find |
| Setpoints, timers, and totalizers | No | Every setpoint had to be re-entered from memory and a screenshot |
| A restore procedure | No | Three people learned it on a weekend |
Firmware is part of the program
A program is written against a firmware version. Newer processors will often accept an older program after the software flashes them, and sometimes they will not, and either way the flashing needs the firmware file and the utility to do it, which the average plant laptop does not have. The spare on the shelf was bought a year ago and has whatever firmware the distributor had that month. The fix is a line in the backup record that names the firmware version the program was tested on, and either a spare flashed to that version when it arrives or the firmware file kept beside the program. Flashing the spare on the day it arrives, then leaving it in the box, turns a weekend into an hour.
The software is part of the backup
Programming software changes, licenses expire, and the vendor eventually stops offering the version that opens an old file. A file that no laptop at the utility can open is not a backup. Keep an installer for the version that matches the program, keep its license or activation details with the same care as the file, and keep one laptop that is known to open every controller on the system. That laptop should not be the one that also does email and gets rebuilt every year. When the software is a subscription, the record needs the account that holds it and who at the utility can log in.
Setpoints and retentive values are not in the file
The program file holds the logic and the initial values the programmer typed in. It does not hold what the operators have set since: level setpoints, alarm limits, timer presets, PID tuning, runtime hours, and totalizers. On many platforms those live in retentive memory in the controller and nowhere else. When the controller dies they die with it, and the restored program comes up with the values of six years ago. The remedy is a periodic upload from the running controller, which captures current values into a file, on a schedule and after any commissioning change. Where the platform separates program and data, save both. Where SCADA holds a copy of the setpoints, make sure it is written down which side is the master and that the SCADA copy is in the SCADA backup.
What a complete backup contains
- The program
- The current uploaded program with comments, tag descriptions, and any add-on instructions or libraries it depends on. Saved with a date and the name of the person who saved it.
- The firmware version
- The exact version the program was tested against, plus the firmware file if the vendor allows it to be kept.
- The hardware configuration
- Rack layout, module part numbers and firmware, I/O addressing, and any module configuration that lives outside the program.
- Communications
- IP addresses, subnet masks, gateways, serial port settings, protocol mappings, and the addresses of every device the controller talks to.
- Data
- Setpoints, alarm limits, tuning, timer presets, and totalizer values, either as a data upload or as a documented list from SCADA.
- The software
- The installer and license for the version that opens the file, and the name of the laptop it is installed on.
- The procedure
- One page that says, in order, how to replace the processor and get the plant back: which file, which laptop, which cable, which address, and who to call.
Where it lives
A network drive is convenient and is exactly where ransomware looks. Keep the working copy on the drive if that is how the plant works, and keep a second copy offline: an external drive in a fire safe, a USB kept off the network, or a cloud copy that the plant network cannot write to without a person doing it. The offline copy is the one that matters when the drive is encrypted, the server is gone, or the building is. Rotate it on the same schedule as the upload, and record the date on the label.
The schedule that survives staff changes
Backups made by one conscientious technician stop when that technician leaves. A schedule written into the maintenance system, with a work order that comes due every quarter and after every change, survives. The work order says what to upload, where to save it, how to name it, and what to compare it against. Quarterly is enough for a plant whose programs rarely change; monthly for one with an active integrator. The task takes fifteen minutes per controller, and it is the fifteen minutes that turns the Friday failure into an afternoon.
The restore test
- 1
Pull the spare and the backup kit
Use the offline copy, not the network drive, so the test proves the copy that matters.
- 2
Flash and configure the spare on the bench
Match the firmware, set the addresses, and load the program with the laptop named in the procedure.
- 3
Compare against the running controller
The software compare function shows every difference. There should be none, or each should be explained.
- 4
Swap it in during a controlled window
With an operator at the plant and the backup control ready, run the process on the spare for an hour. Everything that is going to be wrong shows up in that hour.
- 5
Write it down
Date, who, what failed, what was fixed. Put the spare back on the shelf flashed and labeled with the date it was proven.
A utility that has done this once has a backup. A utility that has not has a file and a hope, and the difference between the two is the weekend.
Frequently asked questions
- The integrator keeps our programs. Is that enough?
- It is a copy, and copies are good, but it is a copy the utility does not control. Integrators change hands, lose staff, and go out of business, and the file they hold may be the one from startup rather than what is running now. Keep your own, and ask for theirs after every change.
- How often should the program be uploaded from the controller?
- After every change, and on a fixed schedule regardless: quarterly for a stable plant, monthly for one where an integrator is active. The schedule catches the online edits nobody wrote down.
- Should we store the backup in the cloud?
- A cloud copy is fine as one of the copies, provided the plant network cannot overwrite or delete it automatically and the account is not shared with the SCADA server. It should not be the only copy; a drive in a safe still works when the internet does not.
- Does the same apply to the HMI and the SCADA server?
- Yes, and more so. SCADA projects, historian configuration, and the operating system image are larger and change more often. They need the same kit: project files, the software version, licenses, and a proven restore, on the same schedule.
Related topics
- PLC Program BackupsWhat a complete controller backup contains, how often to take one, where to keep it, why the upload from the running controller is not the master, and the verification that turns a folder of files into a recovery plan.
- Backup TestingWhy backups that have never been restored fail when needed, and how to find out: spot restores, full restores of a server to an isolated machine, controller program restores to a spare, archive reads, timing each one, and the findings and record that follow.
- Offline CopiesThe backup copy that survives ransomware: why anything reachable over the network will be encrypted too, what offline and immutable mean, options from rotated drives to immutable storage, what goes in the offline set, and the restore test that uses it.
- Retentive MemoryWhat survives a power cycle in a controller and what does not, how retention is backed up by battery, capacitor, or nonvolatile storage, what should be retentive, and how retained data gets lost.
- Firmware ManagementUpdating the firmware of controllers, modules, drives, HMIs, switches, and radios without breaking the plant: why firmware lags and why it still matters, tracking versions, deciding what to update and when, and the update as a planned outage with a rollback.
- Program Will Not DownloadThe software refuses to download, or the download starts and fails. The version, path, mode, memory, and protection problems behind it, and the order to check them in.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.