Richmond study finds transfer-switch failure led to January water plant outage; city adopts corrective actions

Robert Page, an HNTB investigator, told the Richmond City Council’s Organizational Development standing committee that “the root cause of the failure was the failure of the transfer switch and switch gear 6,” and that subsequent electrical and UPS failures allowed filter effluent valves to remain open, flooding the treatment-plant basement.

The HNTB final report, presented at the committee meeting, concluded the flood damaged pumps and critical electrical equipment and caused a nearly 36-hour outage that produced a citywide boil-water advisory. The report attributed the outage to the mechanical failure plus compounding operational issues: undersized uninterruptible power supplies (UPS) with additional loads, delayed or absent staff emergency procedures, limited training for managerial staff, gaps in standard operating procedures (SOPs), delays in capital-project implementation and preventative maintenance, and poor communications during the incident.

The report’s recommended actions aim to reduce risk and improve response. HNTB’s recommendations included implementing seasonal risk assessments that involve all treatment-plant staff; developing SOPs and recurring emergency-response training for treatment-plant managers and operators; evaluating and reprioritizing the Capital Improvement Plan (CIP) and implementation processes; replacing undersized UPS units; adding visual and remote indicators and watertight actuators for filter effluent valves; and creating a crisis-communication plan with clear protocols for notifying wholesale customers and regulators.

Scott Morris, Director of Public Utilities, told the committee many of those corrective measures are already underway. Morris said operations staff have been trained on switch-gear operations and transitioning the plant to generator power, generator automatic transfer switch work is in the CIP, an outside consultant has been engaged to replace undersized UPS units, visual indicators and remote open/close switches for filter effluent valves are being installed, and all filter valve actuators have been confirmed watertight. Morris said staffing adjustments had been made at the plant to add one additional operations person across shifts and two dedicated maintenance staff for day-to-day tasks.

Council members pressed for further detail and accountability. Councilmember Auerbach asked when Hagerty Consulting’s review of the city’s communications response would be available; staff said Hagerty’s report is due May 9 and will focus on the citywide emergency response rather than plant operations. Councilmember Gibson asked why the HNTB document contained only post-event communications and whether pre-event communications were reviewed; HNTB responded their scope focused on the proximate causes of Jan. 6 and that a historic review of all prior communications could be extensive and diffuse their focus.

On costs and next steps, Morris said the city has documented costs to date and that Mayor-referenced estimates of roughly $5 million reflected anticipated additional work; Morris said a firmer interim cost estimate for implementation would be available later in the month and would be distributed to council. Councilmembers asked for a corrective-action matrix with timelines and implementation-cost estimates; Morris said he was developing the matrix and expected to circulate it within days and would include schedule approximations even if full engineering cost estimates required further design work.

Several council members asked whether the outage was avoidable and whether it reflected human error, mechanical failure, or long-running deferred maintenance. HNTB and DPU told the committee the immediate initiating event was an electrical transfer failure caused by the storm; the failure then was compounded by inadequate UPS sizing, the filter-valve behavior and delayed staff response. HNTB said better-prepared staffing, documented SOPs, and training to manually transfer power or operate valves could have reduced the outage’s impacts.

Council discussion also addressed regional coordination. City officials said they have restarted more regular coordination with wholesale water partners and intend to reestablish recurring CIP review meetings so regional customers are aligned on capital needs and cost shares. Officials cautioned that larger structural changes — for example, creating a regional authority — would be a long-term process with significant financial and governance implications.

The committee requested and staff committed to provide: (1) a corrective-action matrix with primary, interim and final actions and target timelines; (2) a clearer interim cost breakdown for repairs and interim measures; (3) continuing updates on coordination with regulators (EPA, VDH) and Hagerty Consulting’s communications assessment; and (4) documentation of SOP and training updates about switch-gear/manual transfer operations. DPU reported some interim operational changes are in place and that capital projects to address the transfer-switch, UPS replacements and valve remote controls are moving forward.

Committee members said they will review the matrix and follow up during upcoming committee meetings and budget deliberations to assess whether requested staffing or capital investments need to be included in the FY26 budget or brought forward as midyear requests.

The report and committee discussion left the city with a mixed immediate picture: a clear mechanical initiating failure compounded by operational and planning gaps. Council members repeatedly requested documentary timelines, cost detail and evidence of training and SOP updates to ensure the corrective measures are tracked and funded.