Computer Model Evaluation

Research Project

Though inherently statistical, model evaluation lacks a unifying statistical framework. NISS was hired to help find a overlying system to help with model evaluation. The research team used Bayesian techniques to measure the degree to which a model captures the underlying reality; theory and methods that allowed dual use of data in both estimation of model inputs and evaluation of outputs. SFCME also involved selection of evaluation functions by which a model and reality are compared. It also looked at design for determining what field or computer simulation data to collect. NISS helped formulate test bed examples (initially, subsurface fluid flow models and traffic simulators). These motivated the formulations and were testing grounds for new theory and methodology. A Virtual Laboratory for Model Evaluation was established to disseminate results, broaden involvement of other researchers (and users), experiment with visualization and other methods in evaluations and created a unique educational and training environment.

This research consisted of two complementary projects. Evaluation of the fidelity of computer models to reality is central to assessing their effectiveness in understanding real phenomena and predicting results of innovative policies.

Names of two projects:

  • Statistical Framework for Evaluation of Complex Computer Models (SFCME)
  • Mathematically/Statistically Based Validation Systems

Technical Report(s):

https://www.niss.org/sites/default/files/pdfs/technicalreports/tr128.pdf

https://www.niss.org/sites/default/files/pdfs/technicalreports/tr162.pdf

https://www.niss.org/sites/default/files/pdfs/technicalreports/tr163.pdf

https://www.niss.org/sites/default/files/pdfs/technicalreports/tr165.pdf

 

Project Goal: 

Does the computer model accurately represent reality? NISS was tasked to find a overlying system to help with model evaluation.

Research Team: 

Principal Investigator(s):  Jerome Sacks, NISS;

Senior Investigator(s): Jim Berger, Duke; M.J. Bayarri, Valencia; J. Cafeo, General Motors; F. Liu, Castilla-La Mancha; C.H. Lin, General Motors; J. Tu, General Motors

Post Doctoral Fellow(s): Jesus Palomo, Rui Paulo, Danny Walsh

Individual Team Members: 
Jerome SacksJesus Palomo