The Intermittent Fault Is a Data Problem

The lift station lost communication for about a minute, once every day or two, at no particular time. The technician had been to the site four times, and each time the radio was fine, the controller was fine, the signal margin was fine, and the site was up. Two radios and a controller had been replaced on the theory that something was marginal. Then someone exported the communication alarm history and sorted it by time of day. Every event fell between 2:05 and 2:15 in the morning, or between 2:05 and 2:15 in the afternoon. The station was not intermittent. It was punctual. It just was not punctual to a clock anyone had been watching.

Why the meter cannot find it

Direct troubleshooting works by observing the fault: put the meter on, see the bad reading, follow it. An intermittent fault is defined by not being there when you look. The technician's visit, the meter, the laptop, and the extra attention are a different set of conditions from the ones the fault happens under, and often the visit itself changes them: the panel door is open, the laptop is drawing the network's attention, the thing that runs at two in the morning is not running. Replacing parts on suspicion, which is what happens when observation fails, is expensive and it teaches nothing, because a fault that stops after a part swap may have stopped for any reason at all. The way through is to stop trying to be there and instead make the system record what was there.

What to capture

Timestamps

Every alarm and event with a time from a synchronized clock. A controller whose clock is twenty minutes off from the SCADA server produces a record that cannot be lined up with anything.

First-out

When several things fail together, which one failed first. SCADA event history at one second resolution usually shows it; a first-out latch in the controller shows it with certainty.

Values around the event

Trend the signals that matter at a fast rate, a sample every second or faster for the analog in question, so the minute before the fault is visible in detail. Historian compression should be turned off for those tags while hunting.

Diagnostic tags

A counter for every occurrence, a latch that holds the last value of an input when the fault bit sets, a timestamp captured in the controller at the moment of the fault, and a rung that records the state of the things you suspect. These take ten minutes to add and are worth a month of visits.

Everything else

A plant log for the period: pump starts, generator exercise, HVAC, backup jobs, radio traffic, weather, deliveries, and who was on site. Half of intermittent faults are correlated with something nobody thought was related.

Reading the pattern

The station that dropped at 2:05 had a cellular router whose data plan renewed its lease every twelve hours, and the renewal took about a minute, during which the tunnel was down. The fix was a configuration change on the router. The two replaced radios and the controller had been fine, and they had been replaced by a good technician doing what troubleshooting by observation would suggest. The record found in an hour what the visits had not found in a month.

Instrumenting a controller for the hunt

1. 1

   Define the fault as a bit

   One rung that sets a bit when the condition you are chasing is true, however it is detected: communication loss, an input that should be on and is not, a value out of range.

2. 2

   Count it

   Increment a retained counter on the rising edge. The count over a week tells you whether the fault is what the operators think it is.

3. 3

   Timestamp it

   Copy the controller clock into retained registers on the rising edge. Keep the last several in a small shift register, so a burst is visible.

4. 4

   Latch the suspects

   On the rising edge, copy the values of the inputs, outputs, and registers you suspect into holding registers. This is the photograph of the moment the fault happened.

5. 5

   Bring it to SCADA

   Put the count, the timestamps, and the latched values on a diagnostic screen and in history. Now the record builds itself while everyone sleeps.

When the record points at hardware

Sometimes the record says the fault is in a cable, a connector, or a terminal, and the pattern says when it happens but not exactly where. That is the point to bring in a recorder: a data logger on the suspect signal at high rate, a port mirror and a packet capture for a network fault, or a temporary transmitter beside the real one. The recorder is placed where the record says to place it, and it is left running until the fault happens, which by now is a predictable event. Measuring is the last step, and it is short, because the data has already done the searching.