Overview
Scientific theories are often translated into statistical models and tested by comparing their predictions with real-world data. Bayesian model comparison can then estimate how much support each model receives.
Led by Paul-Christian Bürkner at the University of Stuttgart’s SimTech Cluster of Excellence, this project addressed a further problem: the probabilities produced by Bayesian model comparison are themselves uncertain. This is known as meta-uncertainty, or uncertainty about uncertainty. The project ran from 2021 to 2024.
Key results
- Developed and evaluated methods for quantifying meta-uncertainty in Bayesian model comparison.
- Used extensive model simulations as an additional source of information. These simulations help reveal hidden assumptions about how well the models represent possible real-world processes.
- Distinguished between a “closed-world” setting, in which the true model is assumed to be among the models being compared, and an “open-world” setting, in which all candidate models may be incomplete or inaccurate.
- Developed neural-network methods for amortized Bayesian inference. These methods perform much of the computational work in advance, making it faster to evaluate complex models on new data.
- Created methods for detecting model misspecification—situations in which a model’s assumptions do not adequately describe the data.
- Produced open-source software and workflows, including JANA and BayesFlow, to make simulation-based Bayesian analysis more efficient and accessible.
Why it matters
Bayesian model comparison can show which theories are better supported by available evidence, but its results depend on assumptions about the data, the models, and the possible alternatives. Quantifying meta-uncertainty makes these limitations more transparent.
The project therefore contributes to more careful and reliable scientific inference. Its methods could improve prediction models in fields such as biology, medicine, climate science, and engineering, particularly when models are complex or computationally expensive.
The results do not identify which theory is objectively true. Rather, they help researchers assess how reliable their model comparisons are and how much confidence they should place in the conclusions drawn from them.
This project produced the following peer-reviewed publications:
- Schmitt M., Pratz V., Köthe U., Bürkner P. C., and Radev S. T. (2024). Consistency Models for Scalable and Fast Simulation-Based Inference. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS). Available on ArXiv.
- Schmitt M., Ivanova D. R., Habermann D., Köthe U., Bürkner P. C., and Radev S. T. (2024). Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inference. Proceedings of the International Conference on Machine Learning (ICML).
- Elsemüller L., Olischläger H., Schmitt M., Bürkner P. C., Köthe U., and Radev S. T. (2024). Sensitivity-Aware Amortized Bayesian Inference. Transactions in Machine Learning Research.
- Schmitt M., Li C., Vehtari A., Acerbi L., Bürkner P. C., and Radev S. T. (2024). Amortized Bayesian Workflow (Extended Abstract). NeurIPS Workshop on Bayesian Decision-Making and Uncertainty.
- Radev S. T., Schmitt M., Pratz V., Picchini U., Köthe U., and Bürkner P. C. (2023). JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models. Uncertainty in Artificial Intelligence (UAI) Conference Proceedings.
- Schmitt M., Radev S. T., and Bürkner P. C. (2023). Meta-Uncertainty in Bayesian Model Comparison. Artificial Intelligence and Statistics (AISTATS) Conference Proceedings.
- Schmitt M., Bürkner P. C., Köthe U., and Radev S. T. (2023). Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks. Proceedings of the German Conference on Pattern Recognition (GCPR).
- Radev S. T., Schmitt M., Schumacher L., Elsemüller L., Pratz V., Schälte Y., Köthe U., and Bürkner P. C. (2023). BayesFlow: Amortized Bayesian Workflows With Neural Networks. Journal of Open Source Software.
