Assembly committee hearing highlights risks of automated decision systems, experts urge testing and oversight

The California State Assembly Committee on Consumer Privacy and Protection heard experts on AI risks and mitigation during an informational hearing that focused first on automated decision systems, their limits and harms. Professor Arvind Narayanan of Princeton University testified that many predictive tools are only modestly better than chance and can entrench bias when used in consequential decisions such as pretrial detention, hiring and medical care.

The panel’s discussion centered on why these systems can fail, how they can compound harms across sectors, and what policymakers should require of public-sector buyers and private vendors. Arvind Narayanan said many automated decision systems rely on past data that “carries the imprint of human biases” and that some tools are “AI snake oil” when presented as precise decision-makers. Alondra Nelson of the Institute for Advanced Study and data scientist Cathy O’Neil urged standards for explanation, contestability and independent testing.

Experts told the committee that two features make automated decision systems especially risky: (1) their tendency to use historical data to predict people’s future outcomes, which imports past inequities into automated…