NISS Ai, Statistics & Data Science in Practice Webinar: Judging the Judges: Statistical Evaluation of LLM-Based Metrics for Trustworthy AI Agents

As AI systems shift from single-shot predictors to multi-step agents, evaluation must evolve from simple accuracy scores to statistically grounded assessments of behavior, reliability, and safety. This talk focuses on the statistics of LLM-based “judges” and multi-metric evaluation pipelines for trustworthy AI agents, drawing on large-scale experiments and tooling built around MLflow and Databricks Mosaic AI. We show how LLM judges are calibrated against human raters using percent agreement and Cohen’s kappa, with confidence intervals computed over diverse academic and proprietary datasets to quantify judgment reliability and avoid over-trusting noisy auto-metrics.

Building on this, we describe a principled evaluation stack that simultaneously measures quality, cost, and latency at both row and run levels, aggregating metrics such as answer correctness, guideline adherence, token counts, and end-to-end latency into statistically interpretable summaries for each experiment. We discuss how scorers and LLM judges are reused consistently from offline testing to online monitoring, enabling hypothesis-driven iteration and drift detection in production.

Finally, we connect these judge-level statistics to broader benchmark design, highlighting recent work on calibrating large benchmark suites (the Mosaic Evaluation Gauntlet) by requiring metrics to exhibit monotonic relationships with model scale, thereby filtering out “noisy” benchmarks that fail basic statistical sanity checks. The talk closes with open problems around uncertainty quantification for judge scores, multiple-testing corrections across large metric suites, and how hybrid models like promptable reward models can unify judgment and reward estimation in a single statistically analyzable framework.