Date: Tuesday, August 26, 2025 - 12:00pm to 1:30pm ET
On August 26, 2025, the National Institute of Statistical Sciences (NISS) hosted the latest session of its AI, Statistics & Data Science in Practice webinar series, continuing its Fall 2025 theme on the critical role of experimental design in AI development. This series highlights how principles of design of experiments and randomization strengthen the reliability, generalizability, and interpretability of AI models. By combining structured experimentation with causal inference methodologies, researchers can improve model robustness whether working in controlled trials or with real-world observational data.
The featured speaker was Dr. Aaditya Ramdas, Associate Professor in the Departments of Statistics and Data Science and Machine Learning at Carnegie Mellon University, whose research spans sequential analysis, causal inference, and methodological innovations for reliable AI. The session was moderated by Dr. Will Wei Sun, Associate Professor of Quantitative Methods at Purdue University’s Daniels School of Business, recognized for his work on statistical foundations of large language models, reinforcement learning, and tensor data analysis. Dr. Ramdas began by outlining the challenge of sequential data analysis, where continuously monitoring experiments can compromise the validity of traditional statistical methods. He introduced Sequential Anytime-Valid Inference (SAVI) methods, which allow for valid inference at arbitrary stopping times, providing flexibility in data collection while preserving statistical rigor. A central tool in this framework is the concept of confidence sequences, which offer simultaneous coverage over time and serve as robust alternatives to traditional confidence intervals.
Building on this foundation, Dr. Ramdas explored applications in sequential causal inference, particularly in both randomized trials and observational studies. He presented new methodologies such as asymptotic confidence sequences and time-uniform central limit theorems, which strengthen inference in dynamic settings where treatments and covariates evolve. He also discussed doubly robust estimation techniques, including sequential cross-fitting, which enable more reliable estimation of average treatment effects across varied experimental designs. Throughout the presentation, Dr. Ramdas emphasized real-world applications, including A/B testing in industry, reinforcement learning, and large-scale online experimentation. He highlighted how sequential experimentation methods address the practical need to manage false discovery rates across hundreds of simultaneous tests in environments like Amazon and Netflix.
Sequential Valid Inference Methods
Aaditya discussed sequential causal inference and the problem of "peeking" at data during ongoing experiments, which can lead to invalid statistical conclusions. He explained that classical p-values and confidence intervals are not suitable for sequential analysis, as they can lead to inflated false positive rates when data is monitored continuously. Aaditya introduced the concept of Sequential Anytime Valid Inference (SAVI) methods, which provide valid statistical inference at arbitrary stopping times, allowing for flexible data analysis without compromising validity. He highlighted the importance of SAVI methods in various fields, particularly in settings without strict oversight, and mentioned ongoing research in this area.
Understanding Confidence Sequences
Aaditya explained the concept of confidence sequences, which provide simultaneous coverage guarantees over time, contrasting them with traditional confidence intervals. He demonstrated examples of confidence sequences for means, quantiles, and covariance matrices, highlighting their wider but more reliable coverage compared to standard intervals. Aaditya also introduced the notion of asymptotic confidence sequences, analogous to the central limit theorem, and mentioned ongoing work in this area.
Sequential Causal Inference Discussion
Aaditya discussed sequential causal inference, focusing on its application in observational settings and its validity across various stopping rules. He clarified that confidence sequences allow for arbitrary stopping rules without compromising validity, and explained their relevance to reinforcement learning and bandit settings. Will managed audience questions, with Aaditya addressing topics such as the applicability of design principles to existing data, the existence of a production version of sequential causal models, and the relationship between confidence intervals, FDR, and FWER. Aaditya noted that some questions would be better addressed in the third part of the talk, and encouraged further questions during the QA session.
Asymptotic Confidence Sequences for Causal Inference
Aaditya introduced the concept of asymptotic confidence sequences (AsympCS) for sequential causal inference, defining them as non-asymptotic confidence sequences that provide almost sure approximations to their oracle counterparts. He explained how these sequences generalize asymptotic confidence intervals and demonstrated a theorem that provides a time-uniform central limit theorem for sequential settings under finite mean and variance assumptions. Aaditya also discussed the role of the tuning parameter rho in choosing the initial peeking time M for the confidence sequence, ensuring efficient use of the error budget after time M. Finally, he outlined how these asymptotic confidence sequences can be applied to doubly robust causal inference in sequential settings, combining insights from Robbins and Robins: the one who defined confidence sequences and the one who made fundamental contributions to semi-parametric inference.
Doubly Robust Estimation Techniques
Aaditya explained the concept of doubly robust estimation for estimating the average treatment effect, introducing the Augmented Inverse Probability Weighted Estimator (AIPW) and the sequential cross-fitting method. He discussed the construction of asymptotic confidence sequences for the average treatment effect in observational studies and randomized experiments, highlighting the importance of accurate estimation of propensity scores and regression functions. Aaditya also addressed the extension of these methods to time-varying treatment effects and other semi-parametric estimation problems, emphasizing the applicability of the classical delta method for such extensions.
Confidence Sequences in Practical Applications
Aaditya presented success stories of implementing confidence sequences in practical applications, including post-election audits and A/B testing at companies like Amazon and Netflix. He explained that while non-asymptotic inference is possible in some cases, asymptotic methods are necessary for doubly-robust causal inference and other settings where bias only vanishes asymptotically. Aaditya also clarified that confidence sequences allow for flexible stopping rules, and the decision to stop an experiment is left to the data scientist based on factors like current estimates, targeted effect size or budget constraints, or any other reason not foreseen in advance.
Doubly Sequential Experimentation
Aaditya discussed the concept of large scale “doubly-sequential” experimentation in the modern information technology industry, where hundreds of new experiments are conducted daily, and the challenge of controlling the false discovery rate (FDR) in an online fashion. He explained that traditional offline FDR methods do not apply to this scenario and introduced the idea of an alpha budget that shrinks as experiments are run and expands as discoveries are made. Aaditya outlined various algorithms developed by his group and others to achieve FDR control across multiple experiments, including the LORD++, SAFFRON and ADDIS algorithms, as well as an online analog of the (e-)Benjamini- Hochberg procedure.
Summary
Aaditya presented on confidence sequences, which are valid at arbitrary stopping times and can be constructed even in non-parametric settings. He explained that these sequences can be obtained by inverting sequential tests and can be combined with online false discovery rate tools to control for multiple testing across a sequence of experiments. Aaditya also discussed the importance of cross-fitting for statistical efficiency and addressed questions about extending the methods to multiple treatments, the relationship to Bayesian methods, and handling high-dimensional data.
The session concluded with an engaging Q&A moderated by Dr. Sun, where audience members explored extensions of these methods to multiple treatments, observational data, and high-dimensional settings. Dr. Ramdas emphasized that sequential causal inference—grounded in careful experimental design—offers a principled way to improve the reliability of AI-driven insights.
NISS extends its sincere thanks to Dr. Aaditya Ramdas for delivering an insightful and thought-provoking presentation, and to Dr. Will Wei Sun for skillfully moderating the session. Their contributions helped make this webinar an engaging and informative part of the AI, Statistics & Data Science in Practice series.