IEEE Systems, Man and Cybernetics Magazine - July 2020 - 21

wireless amplifiers. In particular, the developed prototype
integrates electroencephalogram (EEG), electrocardiogram (ECG), electrooculogram (EOG), and facial electromyogram (EMG) sensors directly into the HMD. We list
potential applications for the instrumented HMD and hope
that the work presented here will inspire researchers from
diverse fields to reproduce and adapt the proposed prototype to instrument their own research and develop nextgeneration immersive applications.
QoE in VR and AR Applications
In the last decade, advances in computational power, computer graphics, and sensors have brought to the market a
diverse array of consumer-grade HMDs for VR and AR
applications. Representative examples include the Oculus
Rift, HTC Vive, PlayStation VR, and Microsoft Hololens, to
name a few. These devices have brought immersive applications to the general public. Moreover, the near future of
VR/AR applications looks promising with the arrival of
improved real-time tracking due to the use of cameras on
HMDs and standalone HMDs, such as the Oculus Quest [1].
In addition to gaming applications aimed at the mass market, AR/VR applications are also burgeoning in areas such
as rehabilitation [2], medical training [3], [4], education [5],
work training [6], and the treatment of different disorders
and phobias [7].
However, despite the advances seen in hardware and
software, VR applications have still not reached credible
simulation of a real experience [8]. First, the sense of
immersion or presence has yet to achieve levels in which
conscious awareness of a simulated environment disappears. Second, the bodily discomfort associated with
exposure to VR content, or so-called cybersickness, can
affect between 30% and 80% of users, with symptoms
lasting for several hours [8]. Moreover, as emphasized in
[9], "ultimately, the success or failure of any system for
immersive communication lies in the quality of the
human experience that it provides, not in the technology
that it uses." QoE is a complex concept that is related to
three groups of influential factors: technological, contextual, and human. Examples of human influential factors
include stress, affective state, engagement, and attention,
among others [10].
Measuring the human perception of immersion, presence, and/or experience as well as cybersickness has
typically relied on subjective testing via (postexperience) questionnaires, such as the Virtual Reality Sickness Questionnaire [11], and the Presence and Immersive
Tendencies questionnaires [12]. Subjective tests are
expensive and time consuming, and they are performed
after the immersive application is finished; therefore,
they rely on the user's ability to recall events and aggregate them into an overall immersion/presence/experience rating. As such, subjective testing can be highly
biased, lacks temporal resolution, and allows for only
offline analyses.
	

As an alternative to subjective methods, recent
research has made use of body-machine interfaces (BMIs)
to characterize the user cognitive state through the analysis of physiological signals [13]-[15]. A special type of BMI
is the passive brain-machine (or brain-computer) interface, which measures neurophysiological signals, usually
EEG and/or near-infrared spectroscopy, and extracts correlates for different perceptual and cognitive processes
[16]. Other BMIs have relied on modalities such as ECG,
respiration, EOG, galvanic skin response (GSR), facial
EMG, eye gaze, and pupillometry, to name a few [15]. BMIs
have been used, for example, to measure stress [17],
engagement [18], emotions [19], sense of presence [8], [20],
immersion [21], experience [22], and cybersickness [8], [23];
thus, they can play a key role in advancing VR applications.
Additionally, multimodal BMIs provide a better understanding of the cognitive process involved, because different physiological modalities exhibit complementary
information [24].
Recently, VR applications have started emerging within
health care [25], neurorehabilitation [26], and educational
programs [5], to name a few domains. In such scenarios, it
is hard to gauge the effectiveness of such interventions in
an objective, quantitative manner, and subjective behavioral assessments or short-/long-term outcomes are typically
monitored (e.g., [27], [28]). Within such scenarios, BMIs
may also play a key role and could provide not only realtime feedback on, for example, student engagement [29] or
stroke recovery [30], but also enable neurofeedback-based
immersive applications [31].
In the past, modalities, such as heart rate (HR) and
GSR, have been used to monitor the affective and stress
levels of users in immersive environments [22]. Although
other modalities, such as EEG, can provide more accurate
measures of, for example, mental workload, the use of
EEG presents many challenges, as an EEG cap needs to be
worn (with dozens of electrodes) under the HMD. This can
be time consuming and messy (if gel-based sensors are
used), and it may cause interference with the collected
EEG signals. The latter condition is exacerbated with
wired EEG systems, as they require the user to remain
seated to minimize movement artifacts; this can be
extremely limiting for mobile immersive applications.
Notwithstanding, Neurable [46] has recently introduced
a six-channel EEG headset integrated into an HMD for
brain-controlled videogames. As the games rely on steadystate, visually evoked potentials (SSVEPs), the EEG channel locations are fixed, thus limiting the use of this device
for applications outside the SSVEP domain. Moreover, only
EEG sensors are integrated, thus further limiting the use of
the device as a passive BMI to explore user cognitive processes. To address these issues, we describe the development of system that combines an open source, wireless,
and multimodal BMI with an off-the-shelf HMD. The proposed system is capable of measuring EEG, EOG, ECG, and
facial EMG in a portable, wireless, and noninvasive manner
Ju ly 2020

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE	

21



IEEE Systems, Man and Cybernetics Magazine - July 2020

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - July 2020

Contents
IEEE Systems, Man and Cybernetics Magazine - July 2020 - Cover1
IEEE Systems, Man and Cybernetics Magazine - July 2020 - Cover2
IEEE Systems, Man and Cybernetics Magazine - July 2020 - Contents
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 2
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 3
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 4
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 5
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 6
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 7
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 8
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 9
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 10
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 11
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 12
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 13
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 14
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 15
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 16
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 17
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 18
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 19
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 20
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 21
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 22
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 23
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 24
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 25
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 26
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 27
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 28
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 29
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 30
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 31
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 32
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 33
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 34
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 35
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 36
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 37
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 38
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 39
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 40
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 41
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 42
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 43
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 44
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 45
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 46
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 47
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 48
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 49
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 50
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 51
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 52
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 53
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 54
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 55
IEEE Systems, Man and Cybernetics Magazine - July 2020 - 56
IEEE Systems, Man and Cybernetics Magazine - July 2020 - Cover3
IEEE Systems, Man and Cybernetics Magazine - July 2020 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/smc_202110
https://www.nxtbook.com/nxtbooks/ieee/smc_202107
https://www.nxtbook.com/nxtbooks/ieee/smc_202104
https://www.nxtbook.com/nxtbooks/ieee/smc_202101
https://www.nxtbook.com/nxtbooks/ieee/smc_202010
https://www.nxtbook.com/nxtbooks/ieee/smc_202007
https://www.nxtbook.com/nxtbooks/ieee/smc_202004
https://www.nxtbook.com/nxtbooks/ieee/smc_202001
https://www.nxtbook.com/nxtbooks/ieee/smc_201910
https://www.nxtbook.com/nxtbooks/ieee/smc_201907
https://www.nxtbook.com/nxtbooks/ieee/smc_201904
https://www.nxtbook.com/nxtbooks/ieee/smc_201901
https://www.nxtbook.com/nxtbooks/ieee/smc_201810
https://www.nxtbook.com/nxtbooks/ieee/smc_201807
https://www.nxtbook.com/nxtbooks/ieee/smc_201804
https://www.nxtbook.com/nxtbooks/ieee/smc_201801
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1017
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0717
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0417
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0117
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1016
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0716
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0416
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0116
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1015
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0715
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0415
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0115
https://www.nxtbookmedia.com