An Analytic Solution to Covariance Propagation in Neural Networks
• SEI Report
Publisher
Software Engineering Institute
Abstract
Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.
Part of a Collection
AI Division Publications
Cite This SEI Report
Wright, O., Nakahira, Y., & Moura, J. (2024, May 2). An Analytic Solution to Covariance Propagation in Neural Networks. Retrieved September 11, 2026, from https://www.sei.cmu.edu/library/analytic-solution-covariance-propagation-neural-networks/.
@techreport{wright_2024,
author={Wright, Oren and Nakahira, Yorie and Moura, José},
title={An Analytic Solution to Covariance Propagation in Neural Networks},
month={May},
year={2024},
institution={Software Engineering Institute, Carnegie Mellon University},
url={https://www.sei.cmu.edu/library/analytic-solution-covariance-propagation-neural-networks/},
note={Accessed: 2026-Sep-11}
}
Wright, Oren, Yorie Nakahira, and José Moura. "An Analytic Solution to Covariance Propagation in Neural Networks." Software Engineering Institute, Carnegie Mellon University. Software Engineering Institute, May 2, 2024. https://www.sei.cmu.edu/library/analytic-solution-covariance-propagation-neural-networks/.
O. Wright, Y. Nakahira, and J. Moura, "An Analytic Solution to Covariance Propagation in Neural Networks," Software Engineering Institute, Carnegie Mellon University. Software Engineering Institute, 2-May-2024 [Online]. Available: https://www.sei.cmu.edu/library/analytic-solution-covariance-propagation-neural-networks/. [Accessed: 11-Sep-2026].
Wright, Oren, Yorie Nakahira, and José Moura. "An Analytic Solution to Covariance Propagation in Neural Networks." Software Engineering Institute, Carnegie Mellon University, Software Engineering Institute, 2 May. 2024. https://www.sei.cmu.edu/library/analytic-solution-covariance-propagation-neural-networks/. Accessed 11 Sep. 2026.
Wright, Oren; Nakahira, Yorie; & Moura, José. An Analytic Solution to Covariance Propagation in Neural Networks. Software Engineering Institute. 2024. https://www.sei.cmu.edu/library/analytic-solution-covariance-propagation-neural-networks/
This content was created for a conference series or symposium and does not necessarily reflect the positions and views of the Software Engineering Institute.