The short answer
Historian Problems
Historian problems fall into a few families: gaps, from a collector that stopped, a buffer that overflowed during an outage, a tag renamed or deleted at the source, a license limit, or a clock jump; wrong values, from a scaling or unit change at the source, a tag pointed at the wrong address, or a change in engineering range; missing detail, from compression and deadband settings coarser than the question being asked; storage growth, from retention, tag count, and collection rate; slow queries, from raw data over long ranges and archives on slow storage; and failed backfill or unreadable archives, from time synchronization and archive file management. Each is located with the collector status, the queue depth, the tag configuration audit, the storage utilization, and the archive file list, and prevented with monitoring and a configuration standard for what is collected and how.
Key points
- A gap is a collector, a buffer, a tag change, a license, or a clock; the gap pattern says which.
- Wrong values in the historian are almost always right values from a changed source; check the tag and the scaling history.
- Detail that was never stored cannot be recovered; set compression for the questions the data must answer.
- Disk growth is retention times tag count times rate; the historian tells you which tag is the largest.
- Backfill depends on time synchronization and buffer sizing at the remote controller.
- Archive files are the data; back them up, register them after a restore, and test a read from an old one.
By symptom
| Symptom | Likely causes | Where to look |
|---|---|---|
| Gap on every tag at once | Collector or historian service stopped; server outage; disk full; license expired | Service status and event log for the period; disk; license |
| Gap on every tag from one site or controller | Communication lost with no store and forward; remote buffer overflowed; backfill failed | Communication alarm history; remote controller buffer configuration; backfill status |
| Gap on one tag | Tag renamed, deleted, or re-addressed at the source; collection disabled; tag beyond the license count | Tag configuration change history; collector tag list; license usage |
| Gap at a fixed time daily or weekly | A scheduled job: backup, archive roll, restart, a maintenance script | Schedules on the server; the job log |
| Gap or overlap of an hour twice a year | Daylight saving time handled in local time instead of universal time | Historian time base setting; server time zone |
| Values wrong by a factor | Scaling or engineering units changed at the source; a range change; a tag pointed at a different register | Source tag configuration and its change history; the date the values changed |
| Values flat where they should move | Compression deadband too wide; a value frozen at the source | Compression settings for the tag; the source |
| Spikes that never happened | Bad quality stored as a number; a communication glitch; a transition captured mid-write of a multi-register value | Quality of the samples; the driver statistics at the time |
| Disk filling faster than planned | Tags added; collection rate increased; compression turned off; retention not enforced | Historian storage report by tag; retention settings |
| Trends slow to open | Raw data over long ranges; archives on slow storage; too many pens; a query without an aggregate | Query type; archive location; client trend configuration |
| Old data unreadable | Archive files moved, deleted, or not registered after a restore; a version change | Archive file list against the storage; the restore procedure |
The checks
- 1
Collector status
Every collector running, connected to its source, and the time of its last write. A stopped collector is the most common cause of a system-wide gap.
- 2
Queue depth
The buffer between the collector and the storage. A queue that grows is storage that cannot keep up or a link to the storage that is slow, and it overflows into a gap.
- 3
Tag audit
The historian tag list against the source tag list: tags renamed, deleted, or re-addressed, and tags added that are not collected. Run it after every source change.
- 4
Configuration
Collection rate, deadband, compression, and retention for the tags in question, against the standard for their class.
- 5
Storage
Free space, the growth rate, and the largest tags. Retention enforced and the purge job running.
- 6
Time
Server, collector, and source clocks synchronized, and the historian storing in universal time.
- 7
Archives
The archive file list, the backup of the archives, and a test read from an archive older than a year.
Detail that was never stored
Compression saves disk by discarding samples that fall within a tolerance of a line between the samples it keeps, and deadband collection discards changes smaller than a threshold. Both are set per tag, and both decide, forever, what questions the data can answer. A level compressed at one percent cannot show a half-percent oscillation; a flow collected at a one-minute average cannot show a ten-second surge. When someone asks the historian for detail it does not have, the answer is not a historian problem but a configuration decision, and the fix is to change the settings for the future and to look elsewhere, the controller or the device, for the past. The configuration standard says what each class of tag needs: fast and raw for control diagnostics, averaged for reporting, and it is reviewed when a new question arrives.
Backfill
A remote controller with a buffer keeps its history while the link is down and sends it when the link returns; the historian then has to accept samples with old timestamps and put them in place. That works when the controller clock is synchronized, the buffer is large enough for the outage, the protocol carries the timestamps, and the historian is configured to accept out-of-order data. Any one of those missing leaves a gap that the site did not have to have. Test it at commissioning by disconnecting the link for an hour and looking at the trend the next day.
Frequently asked questions
- The trend shows a value that changed by a factor of ten on a certain date. Was the historian corrupted?
- Almost certainly not. The source changed: a scaling, a range, an engineering unit, or the tag was re-addressed. Find the change in the source configuration history on that date. The historian stored what it was given; the fix is either a correction factor in reports for the earlier period or, on some historians, a rescaling of the stored values with a record of what was done.
- How much disk should the historian have?
- Bytes per sample times samples per day per tag times tags times retention days, computed from the historian storage report for the actual system rather than from estimates, doubled for growth, plus the archives that are kept beyond the online retention. Then alarm at a threshold that gives months of warning.
- Why do the totals from the historian differ from the totalizer in the controller?
- The historian integrates sampled values; the controller integrates every scan. Sampling, compression, and gaps all reduce the historian total, and bad-quality samples treated as zero reduce it further. Use the controller totalizer for the official total and historize it as a tag; use the historian integral for analysis.
- Can I delete old tags to free space?
- Deleting a tag usually removes its history, which may be a regulatory record. Stop collecting it and leave the history, or archive the tag history to a file before deleting. Check the retention requirement for the data before anything is removed.
Related topics
- Historian Data CollectionDeciding what the historian stores and how often: polled versus exception collection, the deadband per tag, rates for control loops, levels, totals, and compliance values, timestamps at the source, quality codes, calculated tags, and the tag list review that keeps a historian from filling with noise and missing what matters.
- Historian CompressionHow historians store years of data on a disk: exception deadbands at the collector, swinging door compression at the archive, what each discards and what it keeps, the settings that turn a trend into a staircase, lossless alternatives, and how to check that the compressed data still tells the truth.
- Historian ArchitectureHow process history is collected, stored, and served: the collector, the archive, and the client layers, where the historian sits relative to SCADA and the DMZ, single-server and tiered designs for a utility, the store-and-forward buffer that survives outages, and the sizing that decides how many years fit on a disk.
- Long-Term StorageKeeping historian data for years: a retention policy by data class that follows the record rules, tiered storage from online to offline copies, downsampling, archive management, exports that outlive the historian, and the annual proof that old data reads.
- Time SynchronizationWhy every clock in a control system must agree and what happens when they do not: out-of-order events, historian gaps at daylight saving, authentication failures, wrong totals. The time source and hierarchy, how each device is synchronized, and the checks.
- Server FailureHow SCADA servers fail and what to do when one does: failure modes from a full disk to a dead hypervisor host, what the plant experiences with and without redundancy, the recovery sequence from restore to alarm verification, and the monitoring that warns.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.