Date: Tuesday, May 13, 2025 - 12:00pm to 1:30pm
Nancy McMillan introduced our speaker for the session Dr. Amy McGovern, the AI Institute for Research and Trustworthy AI Weather, Climate, and Coastal Oceanography, and the challenges of measuring sharpness in meteorological imagery. They also touched on the importance of validating AI models, the impact of different biases on the reporting of trustworthiness, and the need for a global dataset of observations to improve weather forecasting. Lastly, they discussed the concept of trust in AI, the potential of AI in weather forecasting, and the importance of quantifying AI's impact on weather forecasting. Dr. Amy McGovern, a professor at the University of Oklahoma, was introduced as the speaker for the current webinar, and she encouraged the audience to ask questions throughout the discussion.
AI Institute for Weather Research
Amy discussed the AI Institute for Research and Trustworthy AI Weather, Climate, and Coastal Oceanography, which was funded by NSF in 2020. The institute aims to create new foundational understanding of AI across various use cases, with a focus on weather. The institute is up for renewal this year and has been developing novel, physically-based AI techniques that are demonstrated to be trustworthy. These techniques aim to improve prediction, understanding, and communication of high-impact weather and climate hazards. The institute involves academic institutions, private industry partners, and government and federally funded research centers. The research focuses on developing trustworthy AI methods, including physics-based AI, robust AI, uncertainty quantification, ethical use of AI, and explainable AI. The institute also has a strong community college outreach program and a middle school coding camp to promote STEM careers.
AI in Meteorological Imagery Challenges
Amy discussed the challenges of measuring sharpness in meteorological imagery, particularly in AI-generated images. She highlighted the need for a categorization system for biases in Earth systems applications and presented a paper on this topic. Amy also touched on the importance of data quality, selection, and interpretation in AI models, emphasizing the need for careful consideration of data sources and aggregation methods. She concluded by discussing the potential for AI models to learn physics from data, but raised questions about whether they truly understand the laws of physics.
AI Model Validation and Bias Mitigation
Amy discussed the importance of validating AI models, particularly in the context of weather forecasting, to ensure their trustworthiness and mitigate potential biases. She highlighted the need for understanding the data used in these models and the potential for biases in the data. Amy also mentioned her ongoing work on mitigating these biases and the upcoming start of her postdoc, who will help with this task. Nancy asked about the impact of different biases on the reporting of trustworthiness, to which Amy responded that data bias is likely the biggest issue. Lastly, Amy expressed hope that the recently canceled billion-dollar disaster database would be picked up by another organization.
Weather Prediction Challenges and AI Applications
Amy discussed the challenges of accurately predicting weather, particularly in the context of freezing rain. She highlighted the importance of communicating uncertainty in weather forecasts and introduced a new method for doing so. Amy also presented examples of AI applications in meteorology, including
predicting ocean conditions and estimating visibility from images. She emphasized the need for reliable and trustworthy predictions, especially in areas like Alaska and South Texas, where weather conditions can have significant impacts. Nancy asked about the monitoring of recent cuts to weather prediction impacts, to which Amy responded that there is a decrease in weather forecast prediction performance recently but it is not clear if it is seasonal or due to recent cutbacks in weather balloons, but she is not aware of anyone specifically calculating the impacts of these changes.
Government Data Usage and Sharpness
Amy emphasized the importance of government-funded data, citing a study that showed 90% of the public uses such data daily, but only 10% are aware of it. She used weather data as an example, highlighting the need to communicate the extent of government data usage. Nancy discussed the significance of sharpness in data, particularly in forecasting, as it affects trustworthiness. Amy explained that sharpness is crucial for the realism of features, such as atmospheric rivers, which are essential for accurate predictions. Nancy then raised a question about predicting outside the training data, particularly in regions with high temperatures or limited air conditioning, which could lead to biased data.
Improving Weather Forecasting With Global Data
Amy discussed the need for a global dataset of observations to improve weather forecasting. She mentioned her work on a verification system called Extreme Weather Bench, which aims to make global observations available for extreme events like heatwaves. However, she acknowledged the difficulty in obtaining public health data, such as the number of deaths, due to data protection concerns. Amy also highlighted the development of an open-source community model for global weather forecasting, which is available for use. She ended her overview and asked for guidance on how to proceed with the discussion on trust and AI development lifecycle.
Subjective Trust in AI Context
Amy discussed the concept of trust in AI, emphasizing that it is subjective and context-dependent. She highlighted that trustworthiness is not an objective measure, but rather a result of an individual's experience and varies across individuals. Amy also pointed out that trust in AI is influenced by how it is used, with different levels of importance placed on accuracy depending on the context. She further noted that trust in AI also involves setting policies and regulations to ensure its trustworthiness.
AI in Weather Forecasting: Benefits and Risks
Amy discussed the potential of AI in weather forecasting, highlighting its benefits such as improving performance, bias correction, and providing continuous guidance. She also noted potential drawbacks, including the possibility of over-reliance on AI and lack of hands-on experience. Amy emphasized the importance of trust in AI, suggesting that it is a dynamic process that develops through exposure and interaction with the product. She mentioned ongoing work with the NIST AI Institute to create a weather-based risk management framework to mitigate potential risks. Nancy appreciated Amy's answer, noting the abstract nature of the concept of trust in AI.
AI Trust in Weather Prediction
Amy discussed a recent study on how experts at NOAA evaluate new weather products, particularly those involving AI. The study found that the most important factor in trusting a product was knowledge of its limitations and potential failure conditions. Interestingly, the level of trust did not significantly differ between AI and non-AI methods. The study also looked at the performance of new global AI models in predicting severe weather, with Graphcast consistently rating the highest. Nancy raised the question of how to mathematically reverse engineer the features of these models that relate to trust.
Quantifying AI's Impact on Weather
Amy discussed the importance of quantifying AI's impact on weather forecasting, particularly in extreme weather events. She highlighted the need for standardized data and metrics to compare different events globally. Amy also emphasized the importance of community feedback and the need for trustworthy AI. She shared her work on a new startup, Brightband, which aims to make AI weather forecasting tools available to all. Nancy agreed on the importance of quantifying AI's impact and the need for a strong focus on creating trustworthy AI.
Thanks & Recognition
NISS extends sincere thanks to Amy McGovern for her insightful and engaging presentation, and to Nancy McMillan for skillfully moderating the discussion. Special thanks also go to the organizing committee of the NISS AI, Statistics, and Data Science in Practice series for curating another impactful session. This event would not be possible without the support of our institutional partners and the continued interest of the broader statistical and AI community.
NISS AI Webinar Series Update
The NISS AI statistics and data science in practice Webinar series is set to take a break over the summer and restart in late August. The theme for the upcoming series will be the use of statistics, Statistical design of experiments, and development and refinement of AI systems. Previous events' recordings and slides are available on the NISS Communications YouTube channel and past news stories. Furthermore, we have a JSM session based on the series of webinars, which will be held on Monday, August 4th.