Symptom
A trend shows doubled samples, two values at one time stamp, a sawtooth where the process was smooth, or a totalizer report that counts a period twice; or a period is missing on some tags after a failover, an outage, a rename, or a time change; reports for the period do not reconcile.
The short answer
Duplicate or Missing History
Duplicate history comes from two collectors writing the same tag, a buffer replaying samples the server already had, a time change that made one hour happen twice, or a source that resends its backlog on every reconnect; missing history comes from the same machinery failing the other way, with a buffer that filled or never forwarded, a collector that stopped, or a tag whose history lives under a different name. The two often appear together after the same event, a failover or a long outage, and the historian event log for that period is where the mechanism shows. The cleanup is deleting exact duplicates and merging split names; what is missing is either recovered from a source that recorded it or documented as a gap.
Key points
- Duplicates and gaps usually share a cause: a failover, an outage with a buffer, a rename, or a time change.
- A redundant pair both collecting one tag doubles it; a collector with a replayed buffer doubles the outage period.
- The hour that repeats at the end of daylight saving time is a duplicate by design unless the historian stores in universal time.
- A renamed tag has history in two places; a re-created tag may have none where you look.
- Delete only exact duplicates; recover missing data only from a recorded source; document everything else.
Possible causes and what to check
| Possible cause | What to check |
|---|---|
| Redundant collectors both active | Every sample doubled on tags from one source after a failover or a misconfiguration; both collectors show the tag active. Set the pair so that only one writes, or enable de-duplication. |
| Buffer replayed after reconnect | Duplicates for the outage period only; the collector forwarded its buffer and the server already had samples from another path. Check the buffer and the second path. |
| Source resends backlog on reconnect | A controller or a protocol that sends its event backlog on every connection, and a historian that inserts without checking. Enable duplicate rejection. |
| Daylight saving time change | One hour doubled in the fall and one missing in the spring; the historian storing local time. Store in universal time and display in local. |
| Tag renamed or re-created | History that stops on one name and starts on another; a rebuilt tag with a new identifier. Map or merge the histories. |
| Two tags on one source address | Two tag names collecting one register; the trend of each is complete and the reports double the total. Retire one. |
| Store-and-forward never delivered | A gap for an outage that had a buffer; the buffer filled, the collector disk filled, or the forward failed and the buffer was cleared. Size and monitor the buffer. |
| Manual insert or import run twice | A backfill imported twice; the import log. Delete the second import by its time and tag range. |
| Archive gap | A period with no data on every tag; the archive file for that period was never created, was moved, or is offline. Check the archive list. |
| Clock step on a source | Samples stamped in the future, then samples stamped in the past overlapping them; the time synchronization log. |
Finding the mechanism
- 1
Bound the period
From the raw samples, the exact start and end of the duplicated or missing interval, and the tags affected.
- 2
Match it to an event
The historian log, the collector logs, the redundancy state changes, the communication alarms, and the calendar for a time change, all for that period.
- 3
Look at the sample pattern
Exact duplicates at identical time stamps point at two writers or a replay; offset pairs point at two tags or two clocks; a repeated hour points at the time change.
- 4
Check the writers
How many collectors or interfaces write this tag, whether both members of a redundant pair are active, and whether a second path exists.
- 5
Check the tag history
Renames, re-creations, and address changes on the tag since the period.
- 6
Fix the mechanism first
A cleaned record with the cause still live is dirty again tomorrow.
| Pattern | Mechanism |
|---|---|
| Exact duplicate samples, every tag from one source | Two collectors active for the source |
| Duplicates only during a past outage | Buffer replay onto samples from another path |
| Duplicates for the same hour each autumn | Local time storage across the daylight saving change |
| Two tags with identical values | Two tags on one address; a duplicate tag |
| History under two names | A rename or a re-creation |
| Missing hour each spring | Local time storage; the hour did not exist |
| Missing period on one collector only | That collector, its buffer, or its link |
Cleaning the record
Cleaning is deletion of exact duplicates and merging of split names, with the query used and the counts before and after recorded in the change log. Samples that differ in value at the same time stamp are not duplicates; they are a clock or a source problem, and they are kept until the cause is known. Missing data is recovered only from a source that actually recorded it, a controller log, a local logger, a redundant historian, or a chart recorder, and it is imported with a note of its origin. A gap that has no recorded source stays a gap and is documented.
Preventing it
- Store time stamps in universal time and convert for display; the repeated hour goes away.
- Configure redundant collectors so that one writes and the other stands by, or use a historian that de-duplicates on time stamp and tag.
- Size store-and-forward buffers for the longest outage and monitor their fill level as a tag.
- Treat a tag rename as a change with a history mapping, not as a delete and create.
- Keep one tag per source address, and audit for duplicates in the tag database once a year.
- Synchronize the clocks of every source, collector, and server to one time source, and alarm on drift.
Frequently asked questions
- Is a repeated hour in November a fault?
- It is a consequence of storing local time. The hour between one and two happened twice, and a historian that stores local time either overwrites the first hour or keeps both. Storing in universal time is the fix; until then, reports that cover that night are read with that in mind.
- Can I just delete everything in the period and re-import from the controller?
- Only if the controller recorded the whole period at the resolution the historian needs and the import is documented. Deleting a partially good record to replace it with a partial import trades one problem for another.
- The two values at the same time stamp are different. Which is right?
- Neither is known to be. Two sources with different clocks, or a source that resent a corrected value, produce that pattern. Find the mechanism before choosing; if it cannot be found, keep both, flag the period, and note it in any report.
- Why did the failover duplicate history when it was supposed to protect it?
- Because both members of the pair collected during the transition and both wrote, or the standby forwarded its buffer onto data the primary had already stored. The redundancy design decides who writes and when; test the failover and check the record afterward.
Related topics
- Trend GapsHoles in a trend where data should be: a communication loss, a historian that stopped or fell behind, a buffer that filled or never forwarded, compression set too coarse, a renamed tag, or a clock that jumped. Reading the cause from the shape of the gap.
- Historian ProblemsHow a historian goes wrong and how to find each fault: gaps, wrong values, detail never stored, disks that fill, slow queries, backfill that never arrives, and unreadable archives, with the collector, queue, configuration, and storage checks that locate it.
- Historian ArchitectureHow process history is collected, stored, and served: the collector, the archive, and the client layers, where the historian sits relative to SCADA and the DMZ, single-server and tiered designs for a utility, the store-and-forward buffer that survives outages, and the sizing that decides how many years fit on a disk.
- SCADA RedundancyWhat redundant SCADA servers protect against and what they do not, how failover works for tags, alarms, history, and clients, the network and controller layers beneath it, and why an untested failover is not redundancy.
- Time SynchronizationWhy every clock in a control system must agree and what happens when they do not: out-of-order events, historian gaps at daylight saving, authentication failures, wrong totals. The time source and hierarchy, how each device is synchronized, and the checks.
- Reporting from the HistorianTurning history into the reports a utility must produce: the monthly operating report, compliance reports for the regulator, daily operator summaries, pump run and energy reports, the calculations behind daily minimums, maximums, and totals, handling bad quality and gaps, and building reports once so they run every month without an engineer.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.