The Lift Station That Overflowed With Everything Working

What follows is a composite. No single station did all of this, but every part of it has happened somewhere, and the stations it happened at were not badly built. They had modern controllers, telemetry, generators, and backup floats. They overflowed anyway, and the investigation afterward found nothing broken. That is the point: the failure was in the spaces between the parts.

The night

Rain begins in the evening. By ten the inflow to a duplex station has doubled, and the level control does what it should: the lead pump runs longer each cycle, then the lag pump joins it. At eleven a power blink trips the utility feed for a few seconds, the transfer switch moves to the generator, and both pumps, called by the high level, start on the same second. The generator breaker trips on the inrush. The station is now on neither source. The controller, on its UPS, raises a power failure alarm and a high level alarm and sends both to SCADA over the radio.

At the plant, the alarms arrive on the screen. The operator on shift is in the headworks building dealing with the same storm. The alarm callout system dials the on-call phone for the power failure, a high priority alarm, and reaches a number that belonged to an operator who left in the spring. The high level alarm is configured at the same priority as the power failure; it dials the same number. Twenty minutes later the wet well is at the high float. The backup float circuit, which would run the pumps on floats if the controller were dead, does nothing, because the controller is not dead and the circuit only takes over when the controller watchdog drops. There is no power to run the pumps anyway. At midnight the station overflows into a creek.

What worked

• The level control staged the pumps correctly.
• The transfer switch transferred.
• The controller stayed up on its UPS and generated every alarm it was designed to.
• The radio delivered every alarm.
• The callout system dialed the configured number.
• The backup float circuit was healthy and would have run the pumps if the controller had faulted and power had been present.

What did not exist

A staggered start on generator

The design allowed both pumps to start on the same scan when power returned. A few seconds of delay between them, or a rule that only the lead starts on the first scan after a transfer, keeps a generator that carries two pumps running from tripping on two starts.

A generator sized and tested for the worst case

The generator was sized for the running load and tested monthly with no load. A test under the actual start sequence would have tripped it in daylight.

A callout roster that was maintained

The roster had a number that nobody updated when an operator left, and no escalation to a second person when the first did not acknowledge.

Priorities that meant something

Power failure at a station with a generator is a high alarm; high level at a station with no power is urgent. Configured at the same priority, they dialed the same dead number in the same way.

Response time in the level bands

The high alarm sat a foot below the overflow. At the storm inflow, that foot was fifteen minutes. The band was set from the well geometry years earlier without asking how long it would give.

A second path

When the radio alarm reached a phone nobody answered, there was nothing else: no dialer at the site, no text message to a second person, no alarm on a screen someone was watching.

A backup that covers this case

The backup circuit covered a dead controller. It did not cover a live controller with no power, and nothing did.

The chain

Each item above is a small decision, and each was defensible when it was made. Together they form a chain: a design that assumed power would be there or the generator would carry it, an alarm system that assumed a person would answer, and a level band that assumed time. Break any link and the station does not overflow. The generator carries the load; or the operator is reached; or the high alarm comes early enough for a visit; or a second path reaches a second person. Reliability at a lift station is not the reliability of the parts. It is the number of independent ways the wet well can reach a person with time to act.

Breaking the chain

1. 1

   Calculate the response time

   For each station, the volume between the high alarm and the overflow divided by the design storm inflow, in minutes. If it is less than the time it takes to get someone to the site, lower the high alarm or raise the overflow margin, and write the number on the drawing.

2. 2

   Stagger every start

   On any return of power and on every transfer, one pump at a time with a delay, and on generator power the number of pumps allowed to start together limited to what the generator has been tested to carry.

3. 3

   Test the generator under the real sequence

   A monthly test is a load test with the pumps started the way the controller starts them. A generator that trips in the test is fixed in the test.

4. 4

   Make the callout a maintained system

   A roster that is reviewed when anyone leaves, an escalation to a second and third person, an acknowledgment that stops the escalation, and a monthly test call to every number on it.

5. 5

   Rationalize the priorities

   High level at a station with no power, both pumps failed, and communication loss at a critical station are urgent. Everything else is lower. The urgent ones wake people and escalate; nothing else does.

6. 6

   Add a second path at critical stations

   A cellular text alert from the site controller, an autodialer on the high float, or a second radio path, so that one dead phone number cannot be the whole story.

7. 7

   Cover the no-power case

   A portable generator plan with a receptacle and a person who can be there within the response time; or a permanent generator tested under load; or storage that buys the time.

8. 8

   Test the backup circuit for the case it covers

   Fault the controller with power present and watch the floats run the pumps. Then write down the cases it does not cover, so nobody assumes it does.

Afterward

The investigation at a station like this finds nothing broken, and the temptation is to call it an act of weather and move on. The better outcome is a list like the one above, worked through at every station in the system, because the same chain exists everywhere the same design habits were used. The overflow that gets reported is the one that teaches; the ones that do not happen afterward are the ones that were designed out.