Richmond officials review HNTB after-action report on Jan. 6 water treatment plant outage

The Organizational Development Standing Committee of the Richmond City Council heard the final after-action assessment from HNTB on the Jan. 6 water treatment plant failure, which concluded the outage began with "the failure of the transfer switch in Switchgear 6," causing an electrical transfer failure and a cascade of problems that left the utility offline for nearly 36 hours and prompted a boil-water advisory.

HNTB representative Robert Page told the committee the sequence began when the transfer switch did not move power to the alternate feed, the uninterruptible power supply (UPS) units drained and were unable to close the filter effluent valves. "With the water continuing to flow through the filters, the basement began to flood," Page said, and standby pumps were unable to keep up with the inflow. Flooding damaged pumps and electrical equipment in the basement, which compounded the outage. The report also cited a lack of planning, standards and emergency procedures and weak communications as factors that worsened the event.

The report covered multiple systems and recommended changes across operations and maintenance, staffing, SCADA and power systems, asset management and crisis communications. Page said many recommendations repeat earlier interim findings; new items included a standard operating procedure for manually shutting down the SCADA system to avoid a hard shutdown and a recommendation to reassess UPS sizing. On staffing, HNTB recommended a roster change and an emergency staffing plan; the report specifically advised maintaining four full-time treatment-plant operators on shift rather than the staffing level in place at the time of the event.

Why it matters: Council members pressed for implementation timelines and cost detail, reflecting continuing public concern about service interruptions and the financial and operational steps required to reduce the risk of recurrence. Committee members also sought clarity on who would lead follow-up work and whether proposed changes were in this year's budget.

Key findings and corrective steps presented - Root cause: failure of the transfer switch in Switchgear 6 and subsequent failure of the UPS to maintain control of filter effluent valves, which led to basement flooding and equipment damage. - Systems and process recommendations: seasonal risk assessments, updated standard operating procedures (including for manually shutting down SCADA and manual switching of power feeds), reprioritized capital improvement projects, replacement of undersized UPS units, generator automatic transfer switch (ATS) in an active capital project, remote open/close capability and watertight valve actuators for filter effluent valves. - Staffing and training: HNTB recommended reviewing the organizational structure, implementing recurring formal training for management and operators (including emergency response training) and preparing an emergency staffing plan so leadership and operators can get relief during long events. - Communications: develop a crisis communications plan and notification protocols for internal stakeholders, wholesale customers and regulators; HNTB found city officials were not made aware of the severity of events until about 1 p.m.

Council questions, administration responses and next steps Council members repeatedly asked about the costs and timelines for fixes. Council Member Gibson asked whether the HNTB work overlaps with an assessment by Hagerty Consulting; Interim CAO Sabrina Joy Hogg said the Hagerty communications assessment has a May 9 deadline and is broader, examining the city's overall emergency response rather than only the treatment plant. HNTB and city staff affirmed some overlap but said the two reviews have different scopes.

Director of Public Utilities Scott (Scott) Morris (presenting with HNTB) said DPU has begun implementing corrective actions: operators have been trained in switchgear and generator transitions, DPU engaged another consultant to replace undersized UPS units, a generator ATS is part of a capital project, and DPU is installing visual indicators and remote controls for filter effluent valves. Morris said organization charts for operations and maintenance are under revision and staff augmentation and schedule changes have been piloted to add an additional operator per shift.

On costs, Council members asked about a figure Mayor Vula had referenced of $5,000,000. Scott Morris said the city has incurred many costs and expects more; he said a partial breakdown of interim implementation costs would be available later in the month and the administration would circulate a fuller accounting to council when preliminary engineering permits a more precise estimate. Morris said some long-term costs remain undefined pending engineering and procurement work.

On accountability and prevention, HNTB and DPU staff emphasized a combination of mechanical failure and human factors. "It was a mechanical error that was compounded by a human failure," Page stated. Committee members pushed for documentation: several council members requested a matrix showing each corrective action, a schedule for implementation, and cost estimates; staff said they were preparing that matrix and expected to distribute it to council members shortly.

Regulatory follow-up and regional coordination The presentation noted coordination with Virginia Department of Health (VDH), the EPA and Henrico County utilities. Committee members asked whether VDH or EPA follow-up inspections might require additional work; staff said they expected VDH to pursue a condition assessment through a third party and that any formal findings would include response timeframes (30—2 or 45 days). Members also pressed for renewed regional engagement with wholesale water partners; Morris and Interim CAO Hogg said DPU is reintroducing regular meetings with regional wholesale customers to share CIP planning and oversight.

Votes and committee direction The committee did not take final policy votes on HNTB recommendations. However, members asked staff to produce three deliverables: (1) the corrective-action matrix with timelines and estimated costs, (2) a clearer accounting of costs to date and projected near-term costs, and (3) a schedule for SOP/training rollouts and periodic reporting to council or a standing committee. Staff said they would provide the matrix to council within days and follow up on scheduling and accounting.

Ending: The committee thanked HNTB and DPU staff for the report and work to date and asked that updates appear in upcoming committee meetings and the city's budget briefings so council can track implementation progress and potential budget requests.