Uncertainty Quantification

Monte Carlo (MC) dropout uncertainty quantification (UQ) is performed along with surrogate model-based hyperparameter optimization in HYPPO. MC dropout uses standard dropout, which aims to prevent overfitting in a network. This is achieved by taking each node out of the network with probability \(p\) during training and then scaling the remaining nodes by \(\frac{1}{1-p}\) . MC dropout applies dropout during testing, i.e., when the input is evaluated by forward-propagation, each node is dropped out with probability \(p\) and the output of the remaining nodes is scaled by \(\frac{1}{1-p}\) . Forward-propagating the same input through a network using dropout produces different outputs on each pass, which we use to obtain a measure of variability for the output.

For a more detailed explanation of uncertainty quantification using the MC dropout framework, please refer to Section IV in the HYPPO introductory paper. The following plots can be found in Figure 1 of the paper and can be reproduced using this Google Colab notebook . Refer to the Configuration File page for more information on how to use UQ with HYPPO.

../_images/timeseries.png

Time series prediction of temperature in Melbourne, Australia using MC droput. The yellow and green regions represent \(\pm 1\) and \(\pm 2\) standard deviations from the mean (red line). The cyan lines are the MC dropout predictions. The black dashed lines are mean predictions from five independent trial models.

../_images/classification.png

Image classification for a CIFAR10 image. The confidence interval around the mean probability for each class is shown for a single output image. While the class (Class 8) is correctly identified, the uncertainty of the output probability provides important information regarding the stability/accuracy of the trained model.

PyTorch Implementation

uq_quantifier (trainer[, trials, uq_weights, ...])

Main Uncertainty Quantification (UQ) algorithm.

Main function

hyppo.dnnmodels.pytorch.uq. uq_quantifier ( trainer , trials = None , uq_weights = [0.5, 0.5] , dropout_masks = 30 , data_noise = 0 , device = None , ** kwargs ) [source]

Main Uncertainty Quantification (UQ) algorithm.