Assembly hearing: experts warn automated decision systems can entrench bias; call for audits, procurement rules

Assemblymember Rebecca Bauer‑Kahan, chair of the California State Assembly Committee on Privacy and Consumer Protection, convened an informational hearing on artificial intelligence risks and mitigation where experts urged new rules for automated decision systems, saying the tools are already being used in consequential public‑ and private‑sector decisions and often reproduce or amplify historical bias.

Arvind Narayanan, a professor of computer science at Princeton University, told the committee that “these systems make highly consequential decisions about people that affect our health, employment, education, and even our freedom.” He said two features are common to most automated decision systems: they use predictions about people to make decisions and those predictions are trained on past human behavior, which carries historical bias. Narayanan added that many predictive tools are no more accurate than simple formulas: "criminal risk prediction tools typically have an accuracy of roughly in the ballpark of 70% — which can be matched by a formula with just two variables, the defendant’s age and the number of prior arrests," he said.

Why this matters: witnesses described cases in which automated systems produced harms at scale. Narayanan cited a 2013 Netherlands case in which an algorithm flagged about 30,000 parents for alleged welfare fraud; the resulting enforcement caused financial and emotional ruin for many…