The short answer
Time Synchronization
Every alarm, event, history sample, log entry, and certificate check in a control system depends on a timestamp, and the system works only if the servers, clients, controllers, remote units, network devices, and historian agree on the time. One trusted source, a GPS-disciplined clock or a network time server reached through the DMZ, feeds a hierarchy: the time server synchronizes the hypervisor and the servers, the servers synchronize the clients and the network equipment, and the controllers are synchronized by network time where they support it or by the SCADA writing the time over the protocol where they do not. Data is stored in universal time and displayed in local time so that daylight saving does not produce gaps and overlaps, virtual machines take time from one source rather than two, and every device clock is compared with the source and alarmed on drift.
Key points
- Timestamps are the glue: events, history, logs, and authentication all fail quietly when clocks disagree.
- One source, one hierarchy: the time server, then the servers, then the clients, network gear, and controllers.
- Controllers get time by network time protocol where they can and by a periodic write from the SCADA where they cannot.
- Store in universal time, display in local; daylight saving handled in local time produces an hour of gap and an hour of overlap.
- A virtual machine synchronized by both the hypervisor and a time server fights itself; pick one.
- Compare every clock with the source and alarm on drift; authentication fails at a few minutes of error.
What goes wrong
| Symptom | Cause |
|---|---|
| Alarm from a remote site appears before the event that caused it at the plant | Remote controller clock ahead of the server; timestamps from two sources |
| Historian trend has a gap of an hour in spring and an hour of overlapping data in autumn | Local time storage across a daylight saving change |
| Operators cannot log in; the error mentions time or Kerberos | Client or server clock more than a few minutes from the domain controller |
| Certificate errors on OPC UA or web clients | A clock outside the certificate validity period, often a device reset to its default date |
| Backfilled history lands in the wrong place or is rejected | Remote controller clock wrong while it buffered |
| Daily reports show totals that belong to the wrong day | Report server, historian, and controller disagree on midnight |
| Sequence of events after a trip cannot be reconstructed | Controllers not synchronized; events stamped by server arrival time |
| Security logs from the firewall, the server, and the controllers cannot be correlated | Different time sources or time zones on each |
The hierarchy
| Device class | Synchronized by | Notes |
|---|---|---|
| Time source | A GPS-disciplined clock on the control network, or a network time server in the DMZ that takes time from the internet or the enterprise | Two sources for a large system; the control network never reaches the internet directly for time |
| Hypervisor hosts | Network time protocol from the source | The host provides the hardware clock for every virtual machine |
| SCADA servers and historian | Network time protocol from the source, or from the hypervisor, not both | The server is often the time server for everything below it |
| Clients and workstations | Domain time or network time protocol from the servers | Authentication depends on it |
| Switches, firewalls, radios | Network time protocol from the source | Their logs are useless without it |
| Modern controllers | Network time protocol client in the controller, from the source or the server | Configured once; check that it is enabled |
| Older controllers and remote units | The SCADA writes the time on a schedule: DNP3 time synchronization, or a write to the clock registers over Modbus or the vendor protocol | Daily at least; hourly on units that drift; the write is logged |
| Panel HMIs and standalone devices | Network time protocol where supported; otherwise from the controller or set by hand and checked | The ones set by hand are the ones that are wrong |
Universal and local
Local time changes twice a year where daylight saving is observed, and a historian or a journal that stores local timestamps records an hour that does not exist in spring and an hour that occurs twice in autumn. Trends across those hours show a gap or a fold, reports that sum a day get the wrong day, and events cannot be ordered. The remedy is to store every timestamp in universal time, which never changes, and convert to local time only for display. Every serious historian and SCADA platform supports this; the setting has to be chosen at installation, and changing it later means converting the existing data. Controllers that keep local time are synchronized by the SCADA with the conversion done at the server.
Virtual machines
A virtual machine can take its time from the hypervisor, which is convenient, or from a time server over the network, which is standard. Both at once means two authorities adjusting the same clock, and the result is a clock that jumps. Choose one: usually the hypervisor is synchronized to the source and the virtual machines take time from the network like physical servers, with hypervisor time synchronization disabled for them. A snapshot restored from weeks ago brings its old time with it until synchronization corrects it, which is a reason to synchronize on startup.
Checking
- 1
Compare each clock with the source
Servers, clients, controllers, remote units, network devices, on a list, with the offset recorded. Most platforms can display the controller clock next to the server clock.
- 2
Confirm each device is synchronizing
Not just that it is right now: the time client enabled and pointed at the source, or the SCADA time write scheduled and succeeding.
- 3
Check the time base
Historian, journal, and database storing in universal time, displays showing local, and the server time zone set correctly.
- 4
Look for drift
Trend the offset of the controllers that the SCADA synchronizes; a unit that drifts a minute a day has a failing clock and needs a more frequent write or a repair.
- 5
Alarm
Time source lost, server not synchronized, controller offset above a threshold. These are alarms like any other.
Frequently asked questions
- How accurate does synchronization need to be?
- For alarm ordering across sites and historian consistency, a second or so is adequate; network time protocol delivers far better than that on a local network. For sequence of events within a plant, tens of milliseconds, which network time on a good network provides and which precision time protocol provides where needed. For authentication, the domain tolerates a few minutes, which is why a badly drifted clock shows up as a login failure.
- Can the control network get time from the internet?
- Not directly. A time server in the DMZ can take time from the enterprise or from internet time servers and serve it inward, or a GPS clock on the control network can provide it with no outside connection at all. The firewall rule is one direction, one port, one server.
- The remote unit clock is always wrong on Monday morning. Why?
- The unit loses time over the weekend when something that synchronizes it is not running, or it was power cycled and reset to a default date. Check the synchronization schedule, the unit battery or capacitor that keeps its clock through outages, and the log of time writes.
- Should timestamps come from the controller or the server?
- From the controller when the protocol carries them and the controller is synchronized, because they say when the event happened rather than when the server heard about it. From the server otherwise, with the understanding that the resolution is the poll interval. The alarm philosophy records which applies.
Related topics
- SCADA ServersWhat the servers in a SCADA system do and what a utility should expect of them: the I/O, alarm, historian, and client roles, physical versus virtual, redundancy, sizing, placement, backup, patching, licensing, and monitoring.
- SCADA CommunicationsHow SCADA servers talk to controllers and the field: polling and report by exception, the protocols and what each is for, media from plant Ethernet to radio and cellular, poll rate arithmetic, timeouts and quality, time sync, store and forward, and security.
- DNP3The protocol built for utility telemetry: master and outstation, static and event data classes, unsolicited reporting, time-stamped events, and why it suits slow links that Modbus does not.
- Historian ProblemsHow a historian goes wrong and how to find each fault: gaps, wrong values, detail never stored, disks that fill, slow queries, backfill that never arrives, and unreadable archives, with the collector, queue, configuration, and storage checks that locate it.
- Alarm ServersThe part of SCADA that turns a condition into an alarm and manages its life: where detection happens, the alarm state machine, priority, deadband and delays, acknowledgment across clients, the journal, notification, redundancy, timestamps, and flood behavior.
- Industrial DMZ DesignThe buffer zone between the business network and the control system: what goes in it, the no-direct-path rule, push-not-pull data flows, the firewall pair, and the services a utility actually needs to place there.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.