Instrumentation & Measurement Magazine 24-6 - 92

Table 1 - Methodology characteristics
Wearable devices
Conditions
Reference instrumentation
Population size
Comparison among multiple apps
Performance evaluation
Users' opinion
Free-living
Performance of specific activities (walking, jogging, exercises, etc.)
Gold standard instrument (e.g., electrocardiograph for HR)
Superior class wearable device (e.g., chest-strap devices, considered more
accurate than wrist-worn ones)
1-120 people, young/adults/elderly, healthy/sick
DTC diagnostic apps
Arbitrary selection criteria
Experts' evaluation
Comparison with a standard procedure
Reported only sometimes
Influence of age and familiarity with Internet Reported only sometimes
different parameters to quantify the accuracy of wearable devices:
absolute percentage error, correlation coefficient, bias,
accuracy, precision, etc. Furthermore, the measurement uncertainty
of the device is hardly ever reported. Accuracy depends
on test protocol (activities to be monitored and related intensity,
positioning of the device, in-the-wild or laboratory
setting), subjects' characteristics (skin pigmentation, age,
weight), motion artifacts, sweat, and appropriateness of the
sensor dimension with respect to the subjects' characteristics.
As an example, an error of 5% can be considered acceptable for
HR measurements [16]; however, in the literature, the value of
this error is not always reported in the evaluation of wearable
devices, and when it is, higher values are also found [7].
Concerning DTC diagnostic apps, many studies simply
list the characteristics of the considered apps, without trying
to assess their performance in a more quantitative way. It
is fundamental to have a comparison with a gold standard or
with expert opinions to properly evaluate their performance
and provide the users a global picture of the market offerings
in terms of accuracy and reliability. The technology on which
these apps rely should always be reported; the use of physical
sensors [17] or AI tools surely influence the diagnostic output,
and this information is important to consider [18].
Case Studies
As examples, two case studies are reported here: Apple Watch
for a wearable device and Skin scan for a diagnostic app. As
reported in [7], in the literature Apple watch technology has
been tested according to different protocols, such as walking
test, cycling test, moderate-intensity exercises, and treadmill.
Some report low accuracy in HR measurements, in contrast
with others recording accuracy values up to 99.9%. Moreover,
some researchers consider accuracy and correlation coefficient,
while others report bias, mean or median errors, leading
to inhomogeneous and barely comparable results.
Skin scan app, as reported in [1], has been judged with contrasting
opinions: a study based on 93 pictures [19] processed
92
by the app warned of potential harm linked to misleading results,
while another (with a different evaluation methodology)
found an accuracy of 81% (with a sensitivity and a specificity
equal to 73% and 39.3%, respectively).
Therefore, it is obvious that the testing methodology significantly
affects the results; thus, it is paramount to establish
standard testing protocols to gather comparable data among
different studies and to finally deliver appropriate information
to the user community.
Conclusive Remarks
In general, chest-strap wearables are more accurate than
wrist-worn ones when considering the assessment of HR,
even if they are less practical to wear and consequently less
employed. Regarding step counting, approximately half of
wearable devices show error rates higher than 5% (which is
considered acceptable). Moreover, a better performance of a
device in the assessment of a certain parameter does not mean
a general better performance in measuring other quantities.
The accuracy requirements should be considered, depending
on the featured application of the device: if clinical implications
of the measured data are foreseen, a greater accuracy will
be appropriate; otherwise, for common activity tracking, less
performant devices will be valid.
With regard to DTC diagnostic apps, they should be considered
as a kind of medical device, since they provide a
diagnosis to the user, which should be punctual and accurate
to address the best and safest guidance solution (visit the
emergency room or the doctor's office, self-care, or specialist
medical examination). However, a quantitative evaluation
is rarely reported and even when sensitivity and specificity
are assessed, different methodologies are used, leading to
uneven information. AI technologies can surely improve the
outcomes of general symptom checkers, whereas the use of
sensors (mainly cameras, providing useful pictures for dermatology
apps, for example) can enhance the reliability of
DTC diagnostic apps.
IEEE Instrumentation & Measurement Magazine
September 2021

Instrumentation & Measurement Magazine 24-6

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-6

No label
Instrumentation & Measurement Magazine 24-6 - No label
Instrumentation & Measurement Magazine 24-6 - Cover2
Instrumentation & Measurement Magazine 24-6 - 1
Instrumentation & Measurement Magazine 24-6 - 2
Instrumentation & Measurement Magazine 24-6 - 3
Instrumentation & Measurement Magazine 24-6 - 4
Instrumentation & Measurement Magazine 24-6 - 5
Instrumentation & Measurement Magazine 24-6 - 6
Instrumentation & Measurement Magazine 24-6 - 7
Instrumentation & Measurement Magazine 24-6 - 8
Instrumentation & Measurement Magazine 24-6 - 9
Instrumentation & Measurement Magazine 24-6 - 10
Instrumentation & Measurement Magazine 24-6 - 11
Instrumentation & Measurement Magazine 24-6 - 12
Instrumentation & Measurement Magazine 24-6 - 13
Instrumentation & Measurement Magazine 24-6 - 14
Instrumentation & Measurement Magazine 24-6 - 15
Instrumentation & Measurement Magazine 24-6 - 16
Instrumentation & Measurement Magazine 24-6 - 17
Instrumentation & Measurement Magazine 24-6 - 18
Instrumentation & Measurement Magazine 24-6 - 19
Instrumentation & Measurement Magazine 24-6 - 20
Instrumentation & Measurement Magazine 24-6 - 21
Instrumentation & Measurement Magazine 24-6 - 22
Instrumentation & Measurement Magazine 24-6 - 23
Instrumentation & Measurement Magazine 24-6 - 24
Instrumentation & Measurement Magazine 24-6 - 25
Instrumentation & Measurement Magazine 24-6 - 26
Instrumentation & Measurement Magazine 24-6 - 27
Instrumentation & Measurement Magazine 24-6 - 28
Instrumentation & Measurement Magazine 24-6 - 29
Instrumentation & Measurement Magazine 24-6 - 30
Instrumentation & Measurement Magazine 24-6 - 31
Instrumentation & Measurement Magazine 24-6 - 32
Instrumentation & Measurement Magazine 24-6 - 33
Instrumentation & Measurement Magazine 24-6 - 34
Instrumentation & Measurement Magazine 24-6 - 35
Instrumentation & Measurement Magazine 24-6 - 36
Instrumentation & Measurement Magazine 24-6 - 37
Instrumentation & Measurement Magazine 24-6 - 38
Instrumentation & Measurement Magazine 24-6 - 39
Instrumentation & Measurement Magazine 24-6 - 40
Instrumentation & Measurement Magazine 24-6 - 41
Instrumentation & Measurement Magazine 24-6 - 42
Instrumentation & Measurement Magazine 24-6 - 43
Instrumentation & Measurement Magazine 24-6 - 44
Instrumentation & Measurement Magazine 24-6 - 45
Instrumentation & Measurement Magazine 24-6 - 46
Instrumentation & Measurement Magazine 24-6 - 47
Instrumentation & Measurement Magazine 24-6 - 48
Instrumentation & Measurement Magazine 24-6 - 49
Instrumentation & Measurement Magazine 24-6 - 50
Instrumentation & Measurement Magazine 24-6 - 51
Instrumentation & Measurement Magazine 24-6 - 52
Instrumentation & Measurement Magazine 24-6 - 53
Instrumentation & Measurement Magazine 24-6 - 54
Instrumentation & Measurement Magazine 24-6 - 55
Instrumentation & Measurement Magazine 24-6 - 56
Instrumentation & Measurement Magazine 24-6 - 57
Instrumentation & Measurement Magazine 24-6 - 58
Instrumentation & Measurement Magazine 24-6 - 59
Instrumentation & Measurement Magazine 24-6 - 60
Instrumentation & Measurement Magazine 24-6 - 61
Instrumentation & Measurement Magazine 24-6 - 62
Instrumentation & Measurement Magazine 24-6 - 63
Instrumentation & Measurement Magazine 24-6 - 64
Instrumentation & Measurement Magazine 24-6 - 65
Instrumentation & Measurement Magazine 24-6 - 66
Instrumentation & Measurement Magazine 24-6 - 67
Instrumentation & Measurement Magazine 24-6 - 68
Instrumentation & Measurement Magazine 24-6 - 69
Instrumentation & Measurement Magazine 24-6 - 70
Instrumentation & Measurement Magazine 24-6 - 71
Instrumentation & Measurement Magazine 24-6 - 72
Instrumentation & Measurement Magazine 24-6 - 73
Instrumentation & Measurement Magazine 24-6 - 74
Instrumentation & Measurement Magazine 24-6 - 75
Instrumentation & Measurement Magazine 24-6 - 76
Instrumentation & Measurement Magazine 24-6 - 77
Instrumentation & Measurement Magazine 24-6 - 78
Instrumentation & Measurement Magazine 24-6 - 79
Instrumentation & Measurement Magazine 24-6 - 80
Instrumentation & Measurement Magazine 24-6 - 81
Instrumentation & Measurement Magazine 24-6 - 82
Instrumentation & Measurement Magazine 24-6 - 83
Instrumentation & Measurement Magazine 24-6 - 84
Instrumentation & Measurement Magazine 24-6 - 85
Instrumentation & Measurement Magazine 24-6 - 86
Instrumentation & Measurement Magazine 24-6 - 87
Instrumentation & Measurement Magazine 24-6 - 88
Instrumentation & Measurement Magazine 24-6 - 89
Instrumentation & Measurement Magazine 24-6 - 90
Instrumentation & Measurement Magazine 24-6 - 91
Instrumentation & Measurement Magazine 24-6 - 92
Instrumentation & Measurement Magazine 24-6 - 93
Instrumentation & Measurement Magazine 24-6 - 94
Instrumentation & Measurement Magazine 24-6 - 95
Instrumentation & Measurement Magazine 24-6 - 96
Instrumentation & Measurement Magazine 24-6 - Cover3
Instrumentation & Measurement Magazine 24-6 - Cover4
https://www.nxtbook.com/allen/iamm/26-6
https://www.nxtbook.com/allen/iamm/26-5
https://www.nxtbook.com/allen/iamm/26-4
https://www.nxtbook.com/allen/iamm/26-3
https://www.nxtbook.com/allen/iamm/26-2
https://www.nxtbook.com/allen/iamm/26-1
https://www.nxtbook.com/allen/iamm/25-9
https://www.nxtbook.com/allen/iamm/25-8
https://www.nxtbook.com/allen/iamm/25-7
https://www.nxtbook.com/allen/iamm/25-6
https://www.nxtbook.com/allen/iamm/25-5
https://www.nxtbook.com/allen/iamm/25-4
https://www.nxtbook.com/allen/iamm/25-3
https://www.nxtbook.com/allen/iamm/instrumentation-measurement-magazine-25-2
https://www.nxtbook.com/allen/iamm/25-1
https://www.nxtbook.com/allen/iamm/24-9
https://www.nxtbook.com/allen/iamm/24-7
https://www.nxtbook.com/allen/iamm/24-8
https://www.nxtbook.com/allen/iamm/24-6
https://www.nxtbook.com/allen/iamm/24-5
https://www.nxtbook.com/allen/iamm/24-4
https://www.nxtbook.com/allen/iamm/24-3
https://www.nxtbook.com/allen/iamm/24-2
https://www.nxtbook.com/allen/iamm/24-1
https://www.nxtbook.com/allen/iamm/23-9
https://www.nxtbook.com/allen/iamm/23-8
https://www.nxtbook.com/allen/iamm/23-6
https://www.nxtbook.com/allen/iamm/23-5
https://www.nxtbook.com/allen/iamm/23-2
https://www.nxtbook.com/allen/iamm/23-3
https://www.nxtbook.com/allen/iamm/23-4
https://www.nxtbookmedia.com