Statistical Framework for Evaluation
of Complex Computer Models (SFCME)

Project Summary

Evaluation of the fidelity of computer models to reality is central to assessing their effectiveness in understanding real phenomena and predicting results of innovative policies. Though inherently statistical, model evaluation lacks a unifying statistical framework, which this project will supply.

The foundation of SFCME is quantification -- using Bayesian techniques to measure the degree to which a model captures the underlying reality; theory and methods that allow dual use of data in both estimation of model inputs and evaluation of outputs. Selection of evaluation functions, by which a model and reality are compared, is a central component of SFCME, and essential for finding flaws (causes of "invalidity") in models. Design -- for determining what field or computer simulation data to collect, permeates the framework.

SFCME will be built by focusing on specific formulations of problems. Testbed examples (initially, subsurface fluid flow models and traffic simulators) will motivate the formulations and be testing grounds for new theory and methodology.

A Virtual Laboratory for Model Evaluation will be established to disseminate results, broaden involvement of other researchers (and users), experiment with visualization and other methods in evaluation, and create a unique educational and training environment.

The project team of researchers from NISS, Duke University, and North Carolina State University is experienced in statistics, computation, visualization and the testbed applications. NISS' cross-disciplinary mission and programs provide hooks to adjoin many parts of mathematics, statistics, computation and application areas.

NSF Award Abstract