IEEE Systems, Man and Cybernetics Magazine - July 2020 - 30

[11], [12], which consists of using neuroscience tools and
notably Riemannian geometry-based classifiers (RGCs)
results, e.g., BCIs, to assess the ergonomic qualities of
and convolutional neural networks (CNNs), have shown
interactive systems.
promise for other BCI systems, e.g., motor imagery BCIs.
However, reliably estimating the mental workload from
However, they have not been formally studied and comEEG signals over time, contexts, and subjects is difficult
pared for cognitive or affective states classification.
[4]. For instance, in [4], discriminating low from high workThis article explores such machine-learning algoload in a 2-s epoch of oscillatory EEG activity was possible
rithms, proposes new variants, and benchmarks them with
with a classification accuracy of only about 69%, using a filclassic methods to estimate both mental workload and
ter bank common spatial pattern (FBCSP) algorithm [13]
affective states (valence/arousal) from EEG signals. We
coupled with a linear discriminant
study these approaches with both
analysis (LDA) [14]. In [15], the
subject-specific and subject-in--
authors
used a bilinear CSP and a
dependent calibration to move
RGCs proved to have
linear probabilistic support vector
toward calibration-free systems.
machine (SVM) to classify 1.1-s
Our results suggested that a CNN
the highest mean
epochs of oscillatory EEG activity
obtained the highest mean accuraaccuracy with the
into four levels and obtained 93%
cy, although not significantly so, in
classification accuracy. In [16], the
both conditions for the mental
filter bank tangent
researchers applied an SVM to clasworkload study, followed by RGCs.
space classifier
sify two levels of workload, using
However, the same CNN underperthat we introduce
N-back tasks. They obtained 84%
formed in both conditions for the
classification accuracy on 0.5-1.5-s
emotion data set, which features
in this article.
long periods. In [17], the authors
small training data. In contrast,
extracted features using wavelet
RGCs proved to have the highest
entropy and then applied a multimean accuracy with the filter bank
layer perceptron to classify workload data into seven levels
tangent space classifier (FBTSC) that we introduce in this
with 5-s-long periods. They procured 98% classification
article. Therefore, our results contribute toward improving
accuracy on their own data set and 83% on the data set
the reliability of cognitive and affective states classificafrom [18]. Except for the last study listed, all classification
tion from EEG. They also provide guidelines about when
accuracies were obtained in offline analysis settings.
to use which machine- learning algorithm.
Background
BCI applications enable their users to interact with computers by using brain activity only, usually measured with
EEGs [1]. For example, BCIs can enable people with severe
motor impairment to control a wheelchair with EEG only,
e.g., by imagining left- or right-hand movements to make
the wheelchair turn left or right [2]. Such BCIs are called
active BCIs since users are actively sending commands to
the system [3]. In contrast, the so-called passive BCIs [3]
are not used directly for control, but to monitor users'
mental states in real time and adapt an application accordingly. For instance, passive BCIs were used to estimate
mental workload [4], i.e., the amount of cognitive resources
currently engaged by subjects, and affective states [5], i.e.,
the emotions subjects currently feel.
Mental Workload Estimation From EEG
Passive BCIs were used to study mental workload during
navigation tasks with different input devices [6], during
visualization tasks [7] or plane piloting [8]. Workload estimation was also used to design applications that dynamically adapt to users' states, e.g., video games with adaptive
difficulty [9] or training applications with a sequence of
exercises adapted to the cognitive capabilities of each learner [10]. In general, estimating the mental workload from
EEG is extensively used in the field of neuroergonomics
30	

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE Ju ly 2020

Affective States Estimation From EEG
Passive BCIs estimating affective states are called affective
BCIs (aBCIs) [5]. Examples include studies that found neurophysiological responses to differentiate between frustration and boredom in e-learning [19] and between frustration
and normal game play [20]. Moreover, Rani et al. [21] showed
that both players' enjoyment and skills increased when
tasks were adapted to their affective states rather than to
their performance. In [22], players could modulate their
affective states to influence games' parameters. Automatic
media recommendation [23] or real-world emotions detection [24] can also be used for aBCIs.
Despite an increasing number of aBCIs studies, defining and clustering emotion dimensions remains challenging. There are multiple main approaches to define emotion
classes [5]. The most popular, and the one used in our
study, is the circumplex model of Russell [25], which
assumes that any affective state can be localized on a 2D
plane. The first axis of this plane is the valence, which
ranges from positive to negative feelings, and the second
represents arousal, varying from calm to excited. In [23],
an SVM was used for a three-class problem with 15-s-long
epochs, for valence (low versus neutral versus high, 50%
accuracy) and arousal (low versus neutral versus high,
62%). Valence and arousal (low versus high) were studied
in [26] as well, but with 6-s EEG times. The authors
IEEE Systems, Man and Cybernetics Magazine - July 2020

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - July 2020
