Symptom
The tunnel to a cellular site drops and rebuilds repeatedly, or drops and stays down until the modem is rebooted; polling works for small values and fails for larger reads; the tunnel drops at a regular interval; several sites drop at once; the concentrator log shows authentication or negotiation failures.
The short answer
VPN Tunnel Drops
A VPN tunnel over cellular survives only if both ends agree on how to keep it alive and rebuild it, and the cellular path adds problems a wired tunnel never sees: the modem address can change with every session, so a tunnel that expects a fixed peer address never re-establishes; the carrier translates addresses and drops idle entries after a few minutes, so a tunnel without a short keepalive or dead peer detection goes silent; the two ends rekey on their own lifetimes and a mismatch produces a drop at every rekey; the path carries a smaller maximum packet than the tunnel assumes, so packets above a size are lost and only small traffic gets through; the modem reconnects underneath the tunnel and the tunnel does not notice for minutes; the concentrator hits a session or license limit; and a certificate that has expired or a clock that has drifted fails the authentication on every rebuild. The tunnel logs at the concentrator and the modem, aligned by time, show which, and the fixes are a dynamic-peer configuration or a private static address, keepalives shorter than the translation timeout, matched rekey settings, a clamped maximum segment size, a modem that restarts the tunnel on reconnect, capacity at the concentrator, and valid certificates on synchronized clocks.
Key points
- Read both ends: the concentrator log and the modem log, aligned by time. The reason is in one of them.
- A changing cellular address needs a dynamic-peer tunnel or a private static address.
- Keepalive and dead peer detection shorter than the translation timeout; rekey lifetimes matched at both ends.
- Large packets vanishing is the maximum transmission unit; clamp the segment size.
- Certificates and clocks: an expired certificate or a modem with the wrong time fails every rebuild.
Possible causes and what to check
| Possible cause | What to check |
|---|---|
| Cellular address changes and the tunnel expects a fixed peer | The concentrator log shows the site connecting from a new address and being refused, or the site never reconnecting. A dynamic-peer configuration with identity-based authentication, or a private static address plan. |
| Keepalive or dead peer detection longer than the carrier translation timeout | The tunnel goes silent after a quiet period; the concentrator thinks it is up; nothing passes until a rebuild. Set keepalives to a minute or two. |
| Rekey lifetimes mismatched between the ends | Drops at a regular interval matching one end lifetime; negotiation failures in the log at that moment. Match the lifetimes and the proposals. |
| Maximum transmission unit too large for the path | Small polls work; large reads, file transfers, or web pages through the tunnel hang. Clamp the maximum segment size on the tunnel and reduce the interface unit. |
| Modem reconnects underneath the tunnel | The modem log shows a session drop and reconnect; the tunnel stays down for minutes after. Configure the modem or the router to restart the tunnel on reconnect. |
| Concentrator at a session or license limit | Several sites drop together when others connect; the concentrator log shows limits. Capacity. |
| Certificate expired or clock wrong | Authentication failures in the log; the modem clock far from the concentrator. Renew the certificate; synchronize the modem clock. |
| Proposal or algorithm mismatch after an update | Negotiation failures after a firmware or a concentrator change; the proposals on each end. Align. |
| Firewall between the ends blocking the tunnel ports or protocol | The tunnel never builds from some carriers or after a firewall change; the rules for the tunnel ports and the protocol. Open them. |
| Carrier network address translation behavior changed | Sites on one carrier began dropping on a date; the carrier changed its timeouts or its translation. Shorten keepalives; a private network without translation. |
Read both logs
| Concentrator log | Modem log | Cause |
|---|---|---|
| Site connects from a new address; refused or ignored | Session reconnected with a new address | Fixed-peer tunnel on a dynamic address |
| Tunnel up, no traffic, then rebuild | Nothing | Translation timeout; keepalive too long |
| Negotiation failure at a regular interval | Nothing | Rekey mismatch |
| Tunnel up, some traffic passes | Nothing | Maximum transmission unit |
| Tunnel down for minutes after a site event | Session drop and reconnect | Modem reconnect without tunnel restart |
| Authentication failure | Clock or certificate warning | Certificate or time |
| Limit reached | Nothing | Concentrator capacity |
Settings to align
- Peer identification: by identity or certificate rather than by address, for sites with dynamic addresses; or a private static address on the modem so the address never changes.
- Keepalive and dead peer detection: on both ends, at an interval shorter than the shortest translation timeout in the path, commonly one to two minutes.
- Rekey lifetimes and proposals: identical at both ends, with the rekey initiated by the side that can reach the other.
- Maximum transmission unit: the tunnel interface unit reduced for the overhead and the segment size clamped, so that large packets are split rather than dropped.
- Restart on reconnect: the modem or the site router brings the tunnel down and up when the cellular session changes.
- Certificates: lifetimes tracked, renewed before expiry, and the modem clock synchronized from the concentrator or the network.
Procedure
- 1
Collect
The concentrator log for the site, the modem event and tunnel logs, and the SCADA communication alarms, over a day, aligned.
- 2
Pattern
At a fixed interval; after quiet periods; after modem reconnects; on large transfers only; on all sites of one carrier; after a change.
- 3
Compare the settings
Both ends: peer identification, keepalive, dead peer detection, lifetimes, proposals, unit sizes, certificates, and clocks.
- 4
Test the path
A ping through the tunnel with increasing packet sizes to find the unit limit; a quiet period to find the timeout.
- 5
Change and watch
One setting at a time, both ends together where it applies, and the log for a day.
Private networks
A private carrier network with static private addresses removes the two largest causes: the address never changes, and there is no translation to time out. The tunnel becomes a wired-style tunnel between fixed peers, or is dropped in favor of the carrier private network with the utility own encryption where the design allows. For a fleet of cellular sites it is the arrangement that produces the fewest calls.
Frequently asked questions
- The tunnel drops exactly every eight hours.
- A rekey lifetime of eight hours on one end that the other end does not agree with, or a certificate or a session limit at the concentrator with that period. The concentrator log at the moment of the drop names the negotiation that failed; match the lifetimes.
- Polling works but the historian backfill from the site never completes.
- Large packets are being dropped: the maximum transmission unit. Run the ping test, set the unit and the clamp, and the backfill runs.
- Should the site or the plant initiate the tunnel?
- The site, because the site address may change and the site is behind translation; the plant concentrator has a fixed address and accepts the connection. With a private static address plan either can, and the plant initiating lets it detect a dead site faster.
- How many sites can one concentrator carry?
- As many as its license and its throughput allow, which the vendor states; cellular sites use little bandwidth and the session count is the limit. A concentrator at its limit drops sites when others connect, which looks like a random cellular problem until the log is read.
Related topics
- VPN DesignHow a VPN fits remote access for a control system: it terminates in the DMZ and lands on a jump host, never the control network; site-to-site tunnels carry telemetry; user tunnels require multi-factor and individual accounts; the concentrator is patched fast.
- Frequent ReconnectsA cellular site that connects, drops, and reconnects: a weak signal at the cell edge, tower handoffs, idle session teardown, a NAT timeout that needs a keepalive, a firmware fault, a sagging supply, or an antenna problem. Settings that keep sessions up.
- Data Plan or APN ProblemsA modem that registers but has no data, or runs out mid-month: a wrong or changed APN, a capped or throttled plan, a module on the wrong plan or network, a missing static address, or polling that uses more data than planned. Sizing a plan, cutting traffic.
- Time SynchronizationWhy every clock in a control system must agree and what happens when they do not: out-of-order events, historian gaps at daylight saving, authentication failures, wrong totals. The time source and hierarchy, how each device is synchronized, and the checks.
- How to Diagnose Packet LossFind where and why frames are dropped: measure loss with a continuous ping, localize it hop by hop from both directions, read the port counters on the path, and match the pattern to its cause: a cable, a duplex mismatch, congestion, a loop, or a radio link.
- Segmenting a Remote SiteA lift station, a well, or a tank site as its own zone: the site firewall and what it permits, the modem as transport with no public address, cameras and extras off the control segment, engineering access through the plant, and local control without the link.
Direct contact
Have a controls question?
Reach Eric Sullivan directly about anything on this site, a controls or automation topic, or one of his personal projects.