Whatcom County review finds PSA over‑predicts risk, shows weak accuracy for some outcomes

Whatcom County’s pretrial processes work group heard on Monday that the Public Safety Assessment (PSA) used by the superior court is over‑predicting risk for the county’s pretrial population and shows limited accuracy for some outcomes.

"We find is weak to moderate predictive discrimination," said Dr. Zach Hamilton, a professor of criminology and criminal justice, summarizing his prospective evaluation of 3,551 defendants assessed with the PSA between January 2019 and March 2025. Hamilton said the evaluation paired local PSA scores with outcome data from the Washington State Administrative Office of the Courts (AOC) and looked at three outcomes: failure to appear (FTA), new criminal action (NCA) and new violent criminal action (NBCA).

Hamilton reported an overall AUC (area under the curve) for FTAs of about 0.68, which he described as "moderate" and above commonly cited acceptability thresholds. But he added that two demographic groups — Native American/American Indian and the transcript category labeled "other" — fell below that level. For NCA outcomes the PSA performed worse: Hamilton reported overall discrimination near 0.60 (below the 0.64 standard) and substantial evidence of bias and overclassification for some racial and gender groups. For new violent criminal actions he reported mostly weak results (overall ≈0.62) and cautioned that the low incidence of violent outcomes limits predictive power.

Hamilton said the PSA also showed poor calibration in Whatcom County, meaning predicted risks exceeded observed outcomes. "Your blue line is not there… This is indicating poor calibration," he said, later adding that the calibration error exceeded 33 percent — a level he characterized as poor. In plain terms, Hamilton said, the PSA was identifying some people as higher risk than their subsequent outcomes warranted.

Hamilton told the work group that part of the problem is predictive shrinkage: the PSA was developed on a Kentucky sample and its performance can decline when applied to different populations. He also faulted the local decision matrix that combines FTA and NCA scores into one overall classification, saying the merged approach appears to "muddy the waters" and that the expected stair‑step pattern (low<moderate<high outcome rates) was lacking.

"We would recommend that you recalibrate your tool," Hamilton said, urging the county to adjust item weights and cut points so the PSA better reflects Whatcom County outcomes. He also recommended considering non‑criminal history (dynamic) items — such as employment or housing stability — that can improve prediction and reduce bias, and suggested redesigning the matrix (for example, with a hierarchical classification) rather than merging FTA and NCA into a single decision rule.

During questions, participants raised possible confounders. One attendee noted that bail and release conditions — including monetary bail imposed in addition to conditions — could keep people detained despite a recommended release classification and thus affect observed outcomes. Hamilton said the data he received did not include release condition details and therefore the team could not evaluate how that practice may have influenced results.

Members agreed to schedule a special follow‑up meeting to discuss the report and next steps; staff proposed March 6 for a special meeting and no participant objected. Dr. Hamilton offered to share his slides and to assist with further analysis.

The work group did not vote on policy changes during the meeting; the county’s next actions will be determined at the follow‑up meeting and in subsequent discussions among court staff and the task force.