NIST researcher presents model estimates of vanishingly small false-positive rates for a ballistic matching method, urges more testing

Dr. Forberger, a guest researcher in the surface and nanostructure method metrology group in the engineering physics division at the National Institute of Standards and Technology, presented results and projections meant to quantify false-positive error rates for the congruent matching cells (CMC) technique used to compare breech-face impressions from cartridge cases.

He told the audience that estimating error rates for forensic identification has been a recurring concern in the courts and scientific reviews, citing the Daubert decision and later National Academies and PCAST reports. "Estimating error rate in forensic science identifications has been an issue," he said, framing the talk around how error-rate calculations apply to the CMC method.

The CMC approach divides 3-D topography images of a breech-face impression into cells and counts how many cells are “congruent matching cells” (CMC) under a set of correlation criteria. Forberger said the 3-D topography used in the tests was acquired with a disk-scanning confocal microscope and that similarity is expressed by the number of CMCs that satisfy the criteria.

Using previously published test-fire sets, Forberger reviewed a larger dataset he highlighted: 95 cartridge cases taken from 11 pistol slides (a subset cited from a 2012 study referenced in the presentation). He showed that known matching and known nonmatching pairs in that set form two well-separated histograms and argued that the nonmatching distribution is sufficiently stable to be modeled.

To estimate the chance a nonmatching pair produces many CMCs by random coincidence, Forberger described a binomial model in which the only fitted parameter is p, the per-cell probability that a nonmatching cell pair passes the CMC criterion. He reported a p value of about 0.0011 for the Weller set and used the model to illustrate an integrated false-positive error rate for CMC counts of 21 or higher of roughly "10 to the minus 56," a vanishingly small number in that scenario.

Forberger emphasized these figures are model-based and come from a specific, limited dataset. "The measurement uncertainties, sampling, and model uncertainties are not yet factored in," he said, adding that uncertainty budgets are being developed and that accounting for those uncertainties could increase the computed error rates by orders of magnitude.

He then discussed projections to larger, more varied casework populations, citing colleagues (including Alan Jang) who examined multiple manufactured firearm types. Forberger said the nonmatch distributions in several datasets appear narrow and stable, which he characterized as an optimistic sign that low false-positive rates might persist at scale — but only if databases and testing cover the range of manufacturing methods and class characteristics.

Concluding, Forberger said practical application would require a thorough analysis of all significant sources of uncertainty, a database that catalogs counts of firearms by manufacturing method and class characteristics, and extensive testing across many firearm types. "Extensive testing on different types of firearms fabricated by different methods ... is also extremely important," he said. The presentation closed with his thanks to the audience.

The presentation included an explicit disclaimer that the views expressed are his own and not those of NIST, and he stated he did not endorse any commercial equipment discussed during the methodological description.

What happens next: Forberger recommended broader data collection and uncertainty-budget work before the CMC method’s model-based error-rate estimates are used in casework or offered to courts as demonstrative error-rate evidence.