Assembly hearing: experts warn automated decision systems can entrench bias; call for audits, procurement rules

Assemblymember Rebecca Bauer‑Kahan, chair of the California State Assembly Committee on Privacy and Consumer Protection, convened an informational hearing on artificial intelligence risks and mitigation where experts urged new rules for automated decision systems, saying the tools are already being used in consequential public‑ and private‑sector decisions and often reproduce or amplify historical bias.

Arvind Narayanan, a professor of computer science at Princeton University, told the committee that “these systems make highly consequential decisions about people that affect our health, employment, education, and even our freedom.” He said two features are common to most automated decision systems: they use predictions about people to make decisions and those predictions are trained on past human behavior, which carries historical bias. Narayanan added that many predictive tools are no more accurate than simple formulas: "criminal risk prediction tools typically have an accuracy of roughly in the ballpark of 70% — which can be matched by a formula with just two variables, the defendant’s age and the number of prior arrests," he said.

Why this matters: witnesses described cases in which automated systems produced harms at scale. Narayanan cited a 2013 Netherlands case in which an algorithm flagged about 30,000 parents for alleged welfare fraud; the resulting enforcement caused financial and emotional ruin for many families and led to a national political crisis. He and other panelists also cited examples in the U.S., Canada and Australia where algorithmic tools produced erroneous or discriminatory outcomes in areas ranging from beach‑safety monitoring to job‑screening.

Alondra Nelson, Harold F. Linder Professor at the Institute for Advanced Study and former deputy director for science and society at the White House Office of Science and Technology Policy, framed the problem as a spectrum of algorithmic discrimination. She urged the committee to distinguish types of harms — allocative discrimination (denying access to jobs, credit, housing or health care), surveillance and privacy erosion, targeting and profiling, and cultural misrepresentation — and to focus policy where outcomes are most consequential. Nelson pointed to an IRS process researchers later examined and reported as producing disparate audit rates; she said black taxpayers were estimated to be three to five times more likely to be flagged for audit because an algorithm prioritized "easy audits" tied to the Earned Income Tax Credit and similar patterns.

Cathy O’Neil, a data scientist and founder of an auditing practice for algorithmic systems, urged a practical, measurement‑driven approach she described as designing a “cockpit” for each system. "I design it in three parts and three steps," O’Neil said, describing a process of identifying who could be harmed, developing metrics to measure those harms, and setting acceptable ranges for the metrics. She said regulators and public purchasers can accelerate better practices by requiring standardized tests, phased remediation timelines and, where appropriate, third‑party verification. O’Neil cited a Department of Justice settlement with Meta (Facebook) on housing advertising as an example of a legal resolution that defined an unacceptable disparity and set a remediation path.

Panelists emphasized the role of public procurement to shape markets. Chair Bauer‑Kahan and witnesses described how government contracting requirements — inventories of automated decision systems, explicit effectiveness standards, and procurement preferences for vendors that meet safety and contestability criteria — could create market incentives for safer designs.

Several witnesses described real‑world harms beyond disparate outcomes. Narayanan and Nelson gave examples in health care: a Medicare decision support tool that predicted an 85‑year‑old patient would be ready for discharge in “16.6 days,” after which insurer payments stopped despite the patient’s ongoing severe pain; transcription tools that introduce invented content or racialized stereotypes into medical notes; and monitoring systems that penalize consumers because data from apps and connected vehicles has been sold to insurers. Nelson spoke about Life360 and other location apps and news reporting that linked such data sales to higher auto‑insurance quotes for some users.

Lawmakers’ concerns and public comment: Assemblymembers on the panel raised questions about international competitiveness and costs for California businesses. Some members asked whether heavy regulation would drive companies to other states or countries; witnesses answered that well‑designed rules (for example, phased compliance and clarity about expectations) tend to raise market standards and help good actors. At the end of the hearing, Yvonne Fernandez of the California Federation of Labor Unions urged the committee to bar automated decision systems from making primary employment decisions that affect workers’ livelihoods, saying such decisions should remain with humans.

No formal votes or regulatory actions occurred at the hearing. Committee members said the hearing was part of an ongoing series on AI, and staff and witnesses noted additional hearings and research are planned to clarify costs and implementation timelines.

The committee heard repeated calls from academics and auditors for three practical steps: (1) public‑sector inventories of where government uses automated decision systems; (2) minimum effectiveness, explainability and contestability standards for systems used in high‑stakes contexts; and (3) accessible auditing capacity (both trusted third‑party auditors and procedures allowing independent researchers access to vendor systems).