Assembly committee hearing highlights risks of automated decision systems, experts urge testing and oversight

The California State Assembly Committee on Consumer Privacy and Protection heard experts on AI risks and mitigation during an informational hearing that focused first on automated decision systems, their limits and harms. Professor Arvind Narayanan of Princeton University testified that many predictive tools are only modestly better than chance and can entrench bias when used in consequential decisions such as pretrial detention, hiring and medical care.

The panel’s discussion centered on why these systems can fail, how they can compound harms across sectors, and what policymakers should require of public-sector buyers and private vendors. Arvind Narayanan said many automated decision systems rely on past data that “carries the imprint of human biases” and that some tools are “AI snake oil” when presented as precise decision-makers. Alondra Nelson of the Institute for Advanced Study and data scientist Cathy O’Neil urged standards for explanation, contestability and independent testing.

Experts told the committee that two features make automated decision systems especially risky: (1) their tendency to use historical data to predict people’s future outcomes, which imports past inequities into automated scoring, and (2) the apparent mismatch between vendors’ marketing and real-world performance. Narayanan gave examples including criminal risk scores and hospital-recovery predictions, saying that criminal-risk tools frequently achieve accuracy “in the ballpark of 70%” and that a very simple formula (age and prior arrests) can match much of that performance. He described a Medicare case in which an automated estimate of recovery length led to an insurer stopping payments when a patient had not recovered.

Panelists described different forms of algorithmic discrimination. Alondra Nelson used the Biden administration’s Blueprint for an AI Bill of Rights definition and presented a spectrum that included allocative discrimination (denying access to housing, credit, employment), surveillance and privacy erosion, targeting and profiling (including facial recognition misidentification), and cultural misrepresentation. Nelson cited the Netherlands welfare-algorithm case and the Robodebt controversy as large-scale examples where automated systems produced severe harm. She also referenced reporting showing IRS auditing algorithms disproportionately flagged some low-income taxpayers, which investigators later estimated affected roughly 30,000 parents in the Netherlands example she discussed.

Cathy O’Neil described her auditing practice as building “cockpits” of metrics and limits — identifying who could be harmed, measuring those harms, and setting thresholds at which remedial action is required. She recommended staged oversight and consent decrees as practical enforcement tools, noting the Department of Justice settlement with Meta on housing-related ad targeting as an example where enforced remediation and measurable targets improved outcomes.

Committee members asked about comparative advantages between human decisionmakers and automated tools. Panelists said evidence is mixed: in some settings AI can augment human decisions but in others it degrades outcomes because developers over‑promise “full automation” and the tools are later used without the intended human checks. Narayanan argued that procurement rules and public‑sector inventories of algorithmic systems provide “a leg up” to rights‑respecting vendors and help journalists and researchers evaluate public uses.

The panel also discussed practicality and costs. O’Neil and Narayanan said auditing capacity exists in universities, nonprofits and private firms but that the bottleneck is access: companies must allow researchers and third parties to evaluate models and datasets. They proposed phased compliance windows, third‑party certification when feasible, and regulatory backstops when independent auditors are not yet available.

As the hearing closed, labor representatives in public comment urged that automated decision tools not be permitted to make final employment or discipline decisions affecting workers’ livelihoods without human oversight and recourse. The committee did not take formal votes at the hearing; panelists repeatedly asked the Assembly to consider procurement standards, disclosure requirements for public‑sector deployments, mandatory impact assessments, and enforceable contestability procedures.

Experts recommended incremental, evidence‑focused steps: require inventories of public sector automated decision systems, mandate outcome and impact testing for high‑stakes systems, require explanations and appeal routes for individuals, and use procurement to reward vendors that publish verifiable safety practices. Several witnesses emphasized that these obligations can be phased in so smaller vendors are not unduly burdened while creating a market for trustworthy products.