Instrumentation & Measurement Magazine 24-3 - 27

input for the black-box scenario or feature maps for the graybox scenario. This technique bears similarities to adversarial
training. However, instead of attempting to identify slightly
different inputs that result in a large loss, they perform transformations on the inputs that only marginally modify the
model's behavior.
In their study, Mi et al. [17] consider image processing regression tasks. Hence, they apply input transformations on the
images that do not drastically affect the behavior of the model.
For instance, CNNs tolerate image transformations such as
rotations and flips. Therefore, these are the tolerable perturbations they consider. They were able to achieve uncertainty
estimates that are comparable to those produced by the MonteCarlo dropout method.
For the gray-box scenario, perturbations are not directly introduced at the input level. Instead, they are applied to feature
maps. Hence, they apply evenly distributed Gaussian noise
to feature maps to introduce tolerable perturbation. The noise
is randomly sampled at run time to induce a different output
upon repeating execution of the model on the same inputs.
Moreover, they propose a dropout approach where features
are randomly dropped from the feature maps. Intermediate
layers of CNNs often carry redundant information, hence applying dropout introduces stochasticity without drastically
changing the model's behavior. The ML uncertainty is calculated for both scenarios from the variance in the output that
results from introducing perturbation to the inputs or feature
maps.

[6] JCGM 100:2008, Evaluation of measurement data - Guide to the
expression of uncertainty in measurement, (GUM 1995 with minor
corrections), Joint Committee for Guides in Metrology, 2008.
[7] L. Breiman, " Bagging predictors, " Mach. Learn., vol. 24, pp. 123140, 1996.
[8] C. Szegedy et al., " Going deeper with convolutions, " in Proc. 2015
IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015.
[9] Y. Gal and Z. Ghahramani, " Dropout as a Bayesian
approximation: representing model uncertainty in deep
learning, " in Proc. Int. Conf. Int. Conf. Mach. Learn., pp. 1050-1059,
2016.
[10] M. Teye, H. Azizpour, and K. Smith, " Bayesian uncertainty
estimation for batch normalized deep networks, " in Proc. Int.
Conf. Mach. Learn., pp. 4907-4916, 2018.
[11] S. Ioffe and C. Szegedy, " Batch normalization: accelerating deep
network training by reducing internal covariate shift, " in Proc. Int.
Conf. Mach. Learn., pp. 448-456, 2015.
[12] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, " How does
batch normalization help optimization? " Advances in Neural Info.
Processing Sys., pp. 2483-2493, 2018.
[13] B. Lakshminarayanan, A. Pritzel, and C. Blundell, " Simple
and scalable predictive uncertainty estimation using deep
ensembles, " Advances in Neural Info. Processing Sys., pp. 6402-6413,
2017.
[14] C. Szegedy et al., " Intriguing properties of neural networks, " in
Proc. Int. Conf. Learning Representations, 2014.
[15] I. J. Goodfellow, J. Shlens, and C. Szegedy, " Explaining
and harnessing adversarial examples, " arXiv preprint
arXiv:1412.6572, 2015.

Conclusion

[16] S. Bachstein, " Uncertainty Quantification in Deep Learning, "

The use of ML in I&M will only increase with the advent
of the former. It is therefore crucial to understand how ML
contributes to measurement error and how to quantify its associated uncertainty. The latter subject has only recently been
studied and needs more investigation to provide sufficient
confidence in future ML-based measurement instruments
and methods.

References
[1] M. Vallejo, C. de la Espriella, J. Gómez-Santamaría, A. F. RamírezBarrera, and E. Delgado-Trejos, " Soft metrology based on
machine learning: a review, " Meas. Sci Technol., vol. 31, no. 3, Mar.
2020.

Master's thesis, Universitat Ulm, 2019.
[17] L. Mi, H. Wang, Y. Tian, and N. Shavit, " Training-free uncertainty
estimation for neural networks, " arXiv preprint arXiv:1910.04858,
2019.

Hussein Al Osman (M'12) (Hussein.AlOsman@uottawa.ca)
is currently an Associate Professor with the School of Electrical Engineering and Computer Science, University of Ottawa,
Canada, where he directs the Multimedia Processing and Interaction Group and his research focuses on novel ideas in the
realm of Human Computer Interaction, with particular attention to Applied AI. He received the Ph.D. degree in electrical
engineering from the University of Ottawa in 2014.

[2] M. Myslín, " Machine learning has uncertainty. design for it, "
Towards Data Science, Mar. 16, 2020. [Online]. Available:
https://towardsdatascience.com/machine-learning-hasuncertainty-design-for-it-f015a249a444.
[3] M. Khanafer and S. Shirmohammadi, " Applied AI in
instrumentation and measurement: the deep learning
revolution, " IEEE Instrum. Meas. Mag., vol. 23, no. 6, Sep. 2020.
[4] S. Shirmohammadi and H. Al Osman, " Machine learning
in measurement, part 1: error contribution and terminology
confusion, " IEEE Instrum. Meas. Mag., vol. 24, no. 2, 2021.
[5] A. Ferrero and S. Salicone, " Measurement uncertainty, " IEEE
Instrum. Meas. Mag., vol. 9, no. 3, pp. 44-51, Jun. 2006.

May 2021

Shervin Shirmohammadi (M '04, SM '04, F '17) (shervin@ieee.
org) is currently a Professor with the School of Electrical Engineering and Computer Science, University of Ottawa, Canada,
where he is Director of the Distributed and Collaborative
Virtual Environment Research Laboratory and his research
focuses on multimedia systems and networks, including measurement techniques and applied AI for networking, video
streaming, and health systems. He received his Ph.D. degree in
electrical engineering from University of Ottawa, Canada and
currently serves as the Editor-in-Chief of IEEE Transactions on
Instrumentation and Measurement.

IEEE Instrumentation & Measurement Magazine 27

https://www.towardsdatascience.com/machine-learning-has-uncertainty-design-for-it-f015a249a444 https://www.towardsdatascience.com/machine-learning-has-uncertainty-design-for-it-f015a249a444

Instrumentation & Measurement Magazine 24-3

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-3