Experts urge outcome-focused civil legal aid data, national clearinghouse and DOJ incentives

When Jim Sandman arrived at the Legal Services Corporation (LSC) in 2011, he said grantee reporting required only ‘‘output’’ measures such as cases closed and people served — not outcomes that show what changed for clients.

"Everything that was reported to LSC by the 136 grantees that we had at the time was in the form of output data," said Jim Sandman, a distinguished lecturer and senior consultant at the University of Pennsylvania Law School and president emeritus of the Legal Services Corporation. "I thought we were missing something really important."

Why it matters: Guests on the Talk Justice podcast argued that outcome data — measures such as evictions avoided, dollars of benefits obtained or protection orders secured — better demonstrate the impact of legal aid and help direct scarce resources. Funders increasingly ask for evidence of impact, and programs that lack outcome metrics can struggle to justify priorities.

How to measure it: Logan Cornett, director of research at the Institute for the Advancement of the American Legal System (IAALS), described practical choices and tradeoffs in collecting outcome data. IAALS ran an online survey of more than 10,000 respondents in partnership with Hill of the Netherlands and took steps to limit bias, including using probability-based panels and providing Internet access (hotspots and tablets) to some participants.

"One important consideration is making sure that the vendor that you go with uses a probability sampling or probability-based sampling method," Cornett said, adding that vendors in their study supplied hotspots and tablets to reach respondents without online access.

A complementary approach: David Udell, executive director of the National Center for Access to Justice at Fordham University School of Law, described the Justice Index, which measures policy "inputs" such as the number of civil legal aid attorneys, right-to-counsel statutes and access benchmarks (language access, e-filing, disability accommodations) to compare state policy environments.

"We started not by looking at the substantive law so much, but rather looking at how many attorneys are present for poor people in each state," Udell said, explaining the Index’s focus on legislative and institutional measures that shape outcomes.

Gaps and proposals: Sandman highlighted persistent data shortfalls — no reliable national count of unrepresented litigants, spotty access to real-time court filing data (including evictions), and too few randomized controlled trials testing what interventions work. He suggested one practical lever: condition certain Justice Department grants to state court systems on the collection of standardized data.

"The most efficient way of getting it that I can think of quickly is to require the collection of data as a condition of grants that the Justice Department makes to state court systems," Sandman said, arguing that grant conditions would create incentives for systematic reporting.

Risks and safeguards: Guests warned that metric-driven incentives can be distorted if stakeholders focus on narrow targets. Cornett recommended including people who use the justice system in metric design and collecting qualitative data to capture lived experience alongside quantitative measures.

No policy was adopted on the episode; the discussion offered proposals and best practices rather than binding decisions. The guests agreed on the need for better coordination — for example, a national clearinghouse cataloguing who is collecting what data and how — and on the value of mixing methods: surveys, administrative court data, policy indices and randomized trials where feasible.

The episode is available with show notes pointing to the research discussed. Guest remarks are their own and do not necessarily represent the Legal Services Corporation.