Statistical Strategies Enabling Computer Models
to Meet Scientific, Engineering and Policy Objectives

Lecturers
Jerome Sacks, Duke University
William J. Welch, University of Waterloo

NISS Headquarters, Research Triangle Park, NC
November 7-8, 2002

Application Form

Course Schedule and Outline

Background. The use of computer (simulation) models or “codes” is widespread in science, engineering and even in policy-making. Many of these models are highly complex, slow-to-run and require field data for critical inputs as well as for validating their applicability to real problems. Statistical strategies aimed at treating the complexity (usually appearing in the form of high-dimensional input), the slowness-to-run (preventing more than a scant number of runs), and challenges of combining code output with field data are essential to advancing the effective use of computer models.

Course Description. This short course will provide an overview of experimental design, statistical modeling and analysis strategies that have been developed specifically for treating data from complex codes. Approximations of the code output by statistical surrogates and the visualization of input-output relation-ships are key components of such a strategy. The course will emphasize understanding and portraying the methods through explicit examples; technical issues will be addressed through ancillary material that will be provided.

Lecturers. The course will be presented by Jerome Sacks (Professor, Statistics and Decision Sciences, Duke University and Senior Fellow, NISS) and William J. Welch (Professor, Statistics and Actuarial Science, University of Waterloo). Both were (and are) involved in the early (and continuing) development of statistical methods for computer experiments and their applications in industry and science. The expository article “Design and Analysis of Computer Experiments” (J. Sacks, W. Welch, T. J. Mitchell and H. P. Wynn, Statistical Science 4 409–435) is a useful introduction.

Who Should Attend? The course is meant for both:

• Modelers, engineers and other users of models who are acquainted with basic statistical ideas of correlation and regression (function fitting); and

• Statisticians with interest in applications in which computer models play a crucial role.

Cost. The course is available without charge to NISS Affiliates, who may use their NISS/SAMSI Reimbursement
Accounts to meet expenses. It is open to others, on a space–available basis, at a cost of $1000
per person.

Course Schedule and Outline

Day 1 Morning

1. Introduction

• What is a computer model, and what are its characteristics?
• Why statistics?
• Key issues
• Examples providing context for each; focus on deterministic codes

2. What’s New?

• No replication error; all bias
• ”Space-filling” designs
• Statistical approximation with scant data
• Example: Simplified real example of Highway Design Maintenance code varying only 1 and 2 of more than 35 inputs

3. Random Function Models

• Interpolation methods generated by Gaussian processes
• Connections with Kriging (spatial models)
• Which class of models (which Gaussian processes)?

4. Designs

• Latin hypercube sesigns (LHDs)
• Geometric (space-filling) designs
• Geometric LHDs
• Examples

Day 1 Afternoon

5. Data Adaptive Random Function Model; Assessment of Accuracy

• Likelihood function
• Maximum likelihood estimation
• Cross-validation

6. Detailed Example

• Step-by-step analysis of a real example through cross-validation

7. Visualization

• Analysis of variance (ANOVA)-like decomposition
• Screening; sensitivity
• Return to detailed example

Day 2 Morning

8. Review of Day 1: Case Study Example

9. Optimization

• Step-by-step analysis focused on optimization
• Robust engineering design (Taguchi)

10. Calibration

• Combining data from field and computer model output

11. Validation

• Strategy described via a test-bed example

12. Wrap-up

Day 2 Afternoon

Follow-up Problem Session for attendees with specific models and issues who are interested in follow-up discussion with the lecturers. Those wishing to participate should contact Jerome Sacks (sacks@niss.org; 919–685–9300) beforehand.