The Watchdog Timer

Why a slow scan is dangerous

The controller writes outputs at the end of the scan and reads inputs at the start. While a scan is running, the outputs hold whatever the previous scan set them to and the inputs are ignored. A scan that takes a second instead of twenty milliseconds is a controller that has, for that second, stopped responding to the plant while giving every appearance of running. A scan that never finishes is a controller that has stopped entirely with its outputs frozen in mid-state. The watchdog exists so that neither condition can persist silently.

How it works

The watchdog is a countdown timer set to a configured limit. The controller reloads it at the start of every scan, or every execution of a task on platforms with tasks. If the scan finishes, the timer is reloaded before it expires and nothing happens. If the scan is still running when the timer reaches zero, the controller declares a major fault, stops executing, drives outputs to their configured fault state, and records a watchdog fault against the task. The response is deliberate: a faulted controller with outputs in a known state is safer than a running controller nobody can trust.

What makes a scan run long

• A loop in logic that does not exit: a backward jump whose exit condition never becomes true, or a FOR loop over an array whose length was set wrong.
• A very large instruction: a copy or a fill over a big array, a search through a long table, or string manipulation on every scan.
• A communication instruction that blocks rather than running in the background, on platforms where that is possible.
• A subroutine called far more often than intended, from inside a loop or from multiple places.
• Starvation: higher priority periodic tasks consuming most of the processor, so the continuous task barely runs and its scan stretches.
• A one-time event, a large data transfer or a program upload, that coincides with a heavy scan and pushes it over the limit once.

Setting the limit

The controller reports the last and the maximum scan time of each task. Set the watchdog somewhat above the observed maximum under real conditions, with communications active and SCADA connected, so that normal variation never trips it and a genuine runaway trips it quickly. A limit of several seconds on a task that normally scans in twenty milliseconds means a runaway runs the plant blind for seconds before the fault. A limit one millisecond above the maximum trips on the first busy day.

Finding the cause

1. 1

   Read the fault.

   It names the task and, on most platforms, records the scan time that tripped it. Note both.

2. 2

   Look at what changed.

   A watchdog fault on a program that ran for years follows a change: an online edit, a new routine, a data array grown, a new SCADA client polling hard.

3. 3

   Check the maximum scan time.

   If the maximum sits close to the limit in normal running, the program is heavy and the fault is a matter of time. If the maximum is far below the limit, something specific ran away.

4. 4

   Search for loops and large instructions.

   Backward jumps, FOR loops, copies, searches, and string operations. Ask of each whether its size is bounded and whether it needs to run every scan.

5. 5

   Check the task load.

   On a multitask controller, look at how much processor time the periodic tasks consume. A continuous task that gets ten percent of the processor scans slowly however small it is.

The other watchdog

Inside the processor there is a second, hardware watchdog that the firmware itself must service. If the firmware hangs, the hardware watchdog resets or faults the processor. It has no user setting and it is the reason a controller that has crashed completely still ends up in a defined fault state rather than a frozen one. When a processor faults with a hardware or firmware fault code and no program cause can be found, it is that watchdog that fired, and the processor is usually the problem.