The short answer
The Knowledge That Leaves With the Retiring Technician
When a long-serving controls technician retires, a utility loses the unwritten part of its control system: why the setpoint is where it is, which relay was added and what it bypasses, where the spare radio is, how to get the old software to talk to the old controller, and which alarms mean nothing. None of that was written because the person who knew it was always there. Capturing it takes a deliberate program of documentation, walk-throughs, and recorded explanations, started years before the retirement date.
Key points
- The knowledge at risk is the reasons, the exceptions, and the workarounds; the drawings show what, not why.
- It was never written because it did not need to be while the person was present.
- Capture it with as-built verification, annotated drawings, recorded walk-throughs, and a written list of the things nobody else knows.
- Start years out; a two-week handover at the end captures almost nothing.
- The permanent fix is a system that does not depend on anyone: narratives, backups, standards, and a second person on everything.
Every utility has one. The technician who has been there twenty-five years, who knows which lift station has the relay that was added in 2009 and why, who can get the old programming software to run on the laptop nobody else can log into, who knows that the high level alarm at the west tank is a float that sticks and the one that matters is the transmitter. That person is retiring, and the industry is retiring them by the thousand. The control systems they maintained were built, in ways nobody quite intended, on the assumption that they would always be there.
What is actually lost
The drawings show what is connected. The program shows what the logic does. What neither shows is why: why the lag pump starts a foot higher at station six than anywhere else, why the ferric feed is paced to raw water flow rather than influent, why the dissolved oxygen setpoint in basin two is lower, why the alarm from the generator at the plant is ignored on Tuesdays. Some of those reasons are good ones, decided in a meeting fifteen years ago. Some are workarounds for a problem that was fixed a decade later. Nobody can tell which without the person who was there, and a new engineer who removes the wrong one learns why the hard way.
| Category | Examples | Where it lives now |
|---|---|---|
| Reasons for settings | Setpoints, bands, delays, and the events that set them | Memory |
| Undocumented changes | Relays added, wires moved, logic patched during an outage | The panel, unmarked; memory |
| Workarounds | The valve that must be opened by hand first; the reset that needs two tries | Memory |
| Tools and access | Old software versions, license keys, cables, passwords | A laptop, a drawer, memory |
| Spares | Which shelf, which box, which ones are actually bad | Memory |
| Vendor relationships | Who to call, who actually answers | A phone |
| History | What failed before, how it was found, what fixed it | Memory |
| Judgment | Which alarms matter; what normal looks like on each trend | Memory |
Why it was never written
It was not negligence. Writing down why a setpoint is where it is takes time, and the utility never needed it written because the technician could be asked. Documentation systems reward the drawing that is required for the project and not the note that explains the exception made afterward. And a great deal of the knowledge is not the kind that fits on a form: it is the sense that the trend looks wrong today, built from thousands of days of looking. The person did not hoard it. The organization never built a place to put it.
A capture program
- 1
Start early
Three years before the expected retirement, not three weeks. Knowledge comes out in the course of work, not in an interview.
- 2
Verify the as-builts
Walk every panel with the technician and the drawings, and mark every difference. The relay that is not on the drawing gets drawn and gets a note that says why it exists.
- 3
Annotate the settings
Export every setpoint, band, delay, and alarm limit from every controller and SCADA, and go through them with the technician, writing the reason beside each one that has a story. The ones with no story are candidates for review.
- 4
Record the walk-throughs
A phone camera and a site visit. The technician explains each site as they would to a new hire: what it does, what fails, where things are. The video goes in the engineering library. It is imperfect and it is far better than nothing.
- 5
List the things nobody else knows
Ask directly: what would break if you were gone tomorrow. Software, passwords, cables, vendor contacts, the box of spares in the back of the truck. Write the list, then fix each item so that it is no longer true.
- 6
Pair them
A second person on every site visit and every change for the last years, so that the judgment transfers by practice. It is the only way the trend sense moves.
- 7
Fix the workarounds
Every workaround captured is a defect. Fix what can be fixed, so that the knowledge needed to operate the plant shrinks.
The permanent fix
Capturing what one person knows is triage. The durable fix is a control system that does not depend on anyone: a control narrative that says what the system does and why, drawings that match the panels and are revised with every change, program backups with the software and licenses to use them, standards that make every site alike so there is less to know, a credential vault instead of a memory, and two people who can do every job. Utilities that have those find that the retirement is an event rather than a crisis. Utilities that do not are one retirement, or one accident, from a control system nobody understands.
The other side
The technician who is leaving usually wants this to go well. They built the system, they are proud of it, and they know better than anyone what will happen if the knowledge goes with them. What they need is time set aside to do the capture instead of being asked to fit it around the work, a person to hand it to, and the sense that the organization values it. A utility that gives them that gets far more than a binder.
Frequently asked questions
- We cannot afford a second person on everything.
- A utility that cannot afford a second person can afford a narrative, verified drawings, backups, and a credential vault, which cover most of the risk. The second person is the remaining part, and a shared technician with a neighboring utility is one way small systems get there.
- Can an integrator capture this instead?
- An integrator can verify as-builts and write a narrative from the program, which is valuable. The reasons and the judgment come only from the person who has them, and the integrator would have to interview them anyway.
- What if the technician already left?
- Do the as-built verification and the settings review without them, treat every unexplained setting and relay as a question, and call them. Most retired technicians will spend an afternoon on the phone for the plant they built.
- How do we know the capture worked?
- Have the successor run a site alone for a month before the retirement and keep a list of every question they had to ask. Each question is a gap; fill them while the answer is still in the building.
Related topics
- Control NarrativesWhat a control narrative is, what it must contain, how to write statements that can be tested, and how it drives programming, FAT, and the owner’s acceptance.
- PLC Program BackupsWhat a complete controller backup contains, how often to take one, where to keep it, why the upload from the running controller is not the master, and the verification that turns a folder of files into a recovery plan.
- Legacy SystemsKeeping older controllers running and planning a migration: the families still found in water plants, the risks that grow each year, software to keep alive, batteries and memory, tested spares, the security problem, and a migration the plant can survive.
- Control SchematicsThe ladder-style drawing of every control circuit in a panel: how a schematic is organized by line number, the conventions for contacts, coils, and cross-references, what belongs on a schematic and what belongs on a wiring diagram, and reading one under pressure.
- Default CredentialsThe passwords printed in the manual: where they hide in a control system, why they are the first thing an attacker tries, how to find every one on site, and the procedure for changing them without locking yourself out.
- The Control Narrative Is the ContractMost control system disputes, delays, and disappointments trace to one missing document: a narrative that says, in plain language, and how it keeps paying after startup.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.