Panel: practical testing, simple rubrics and monitoring keep legal AI safer and more useful

Quentin Stenhouse, of Lehi Legal Consulting and the Suffolk University Law School Legal Innovation and Technology Lab, opened a conference session on testing AI for legal services, saying teams need clear goals and practical test plans before investing in production systems. "We want to avoid harm to clients at all costs," Margaret Hagen of Stanford Law School said, urging a low risk tolerance for legal applications.

Panelists presented case studies showing how focused evaluation and monitoring changed real projects. Stenhouse described an intake classifier built to route plain‑language user descriptions to existing lawyer referral categories in Oregon. He said the team used labeled queries provided by the Oregon State Bar and ran roughly 400–420 curated examples through different methods: a classical model called Spot (~60% accuracy), simple keyword matching (~54% accuracy), and an ensemble voting approach that proved both more accurate and about one‑third cheaper than using the best single model alone.

Drew Jackson, of the People’s Law School, reviewed his organization’s Beagle Plus chatbot, a retrieval‑augmented system launched in February 2024. Jackson said Beagle Plus has handled more than 15,000 conversations and that 82% of respondents judged the tool helpful; after moving parts of the stack to newer models, he reported helpfulness ratings rose to about 90% on roughly 1,000 recent data points. "We monitor all the conversations," Jackson said, and described using Langfuse for ongoing monitoring paired with a small, shared team that reviews flagged interactions (about 0.4 full‑time equivalent).

All three panelists emphasized two compact rubric dimensions—safety and value—rather than sprawling scorecards. Safety was defined as providing accurate information that does not mislead a user about rights or available steps; value meant providing practical, jurisdiction‑specific guidance that empowers users. Stenhouse and Jackson described an iterative workflow: build a small, representative test set, run experiments across model and prompt choices, use an automated scoring pipeline (an "LLM as judge") to surface likely regressions or severe errors, then verify with human reviewers before deploying changes.

Panelists also tackled common operational issues. On the risk that public benchmark datasets could be scraped and later absorbed into large model training runs—rendering them poor independent tests—they advised several mitigations: keep core test sets private when feasible, use holdout or cross‑jurisdiction samples for validation, generate synthetic test items to expand coverage, and rely on monitoring to detect drift. "If your dataset is public, there is a lag before it becomes training material," Stenhouse said, recommending cross‑state sample sharing and synthetic augmentation when necessary.

When asked about evaluating multi‑turn conversations (multi‑exchange dialogs), the panel said there is no easy, off‑the‑shelf solution yet. Approaches include AI‑generated multi‑turn simulations, breaking narratives into factual elements (a "digital twin") and scoring retrieved facts against expected follow‑ups, but these methods require more engineering and human oversight than single‑turn evaluation.

Panelists closed with pragmatic advice: start small, involve users and content experts in rubric design, focus testing on a few meaningful model configurations, and scale monitoring intelligently (prioritizing high‑severity cases). Materials and test datasets referenced in the session are available on JusticeBench.org and in the conference app; the presenters also encouraged attendance at a hands‑on testing workshop scheduled for the following day.

The session ended with a Q&A and an offer to continue technical follow‑up with the panelists and their tech partners.