IEEE Systems, Man and Cybernetics Magazine - July 2020 - 15

BMI decoder being outdated: classification accuracy or
application-dependent metrics will be unable to identify
these different situations. Consequently, also, a reported
increase or an overall "learning curve" on such metrics
cannot be used to claim subject-learning effects beyond a
reasonable doubt, certainly not in the case where parallel
interventions on the BMI algorithm (i.e., periodic or realtime recalibration or feature reselection) may account for
the performance enhancement. Our previous work provides evidence of this potential mismatch [42]: the
increase in classification accuracy exhibited by the majority of 10 participants during spelling with an online adaptive SMR-based brain-computer interface (BCI) was not
followed by the anticipated strengthening in the separability of their brain patterns. Conversely, an average discriminancy decrease was observed. In that case, we were able
to show that performance boosting was solely attributable
to better-fitted classifiers thanks to their adaptation.
In another of our studies [49], several end users exhibited, on average, improvements in BCI command accuracy
over a maximum of 10 sessions before proceeding to control a telepresence robot. However, further inspection
revealed that this increase did not correlate with enhancement in the discriminancy of the power spectral density
(PSD) features used in the decoders and seemed to mainly
derive from better parametrization of the BMI telepresence device and the BMI hyperparameters. It is noteworthy to mention that, for the exact same reasons, metrics
like classification accuracy are insufficient to isolate
improvements on the machine-learning side. To address
this problem, we have previously introduced a "classifierprecision" metric, suitable for decoders that belong to or
can be represented by generative modeling [42].
Using classification accuracy or regression fitness metrics has prevailed in the BMI field following the introduction of machine learning. Most machine learning is
grounded on assumptions like stationarity and independent identically distributed data. When these assumptions
hold, an increase in performance of a fixed, trained model
seems reasonable to be attributed to the underlying classdependent distributions becoming more separable; however, neural signals (and the features computed on those) are
notorious for violating such assumptions. Nonstationarity
effects have been well described [16], [37], [42]. A violation
of the independence assumption and a potentially varying
degree of it over time may invalidate classification performance estimation through techniques like cross validation
and training-testing split [72], explaining potential "spurious" performance improvements that, in fact, do not represent subject learning.
The "identically distributed samples" assumption may
also be violated when subjects (often, subconsciously)
employ different mental strategies over time, which can be
viewed as another case of nonstationarity not necessarily
manifesting simply as "shifts" in the feature space [37].
Importantly, even with a fixed model, the estimation of
	

BMI Control
Co Devi
nfig ce
ura
ization
tion
Custom
Ma
Le chin
n
arn e
Selectio
ing
Feature timation
s
rE
Su
Decode
Le bjec
arn t
ing
ility
Separab

Figure 1. The pyramidal nature of BMI control and

coadaptation.

metrics like classification accuracy is known to be sensitive to a number of factors such as the class-wise balance
of samples, the cardinality of the data set, and the tendency of the used model to overfit [73]. Therefore, there is still
no guarantee that an increase in accuracy (especially if
small in magnitude, as it is usually the case) actually corresponds to better brain signal modulation.
Given that the common machine-learning evaluation
metrics employed are insufficient to assess the existence
and magnitude of subject-learning effects, we advocate the
use of metrics that directly measure improvements in
brain activity modulation, i.e., metrics directly computed
at the feature level. Additionally, to uphold the operant
conditioning nature of subject learning still thought to be
the dominant learning model in BMI, these feature-based
metrics should be computed either on the same features
giving rise to BMI control or, at least, on directly dependent ones. Indeed, the theory of instrumental learning dictates that learned regulation should necessarily reflect the
variable fed back to the user during closed-loop control;
though, one cannot exclude the case that a change could
occur in a related physiological variable, e.g., an increase
in brain connectivity as a result of SMR-based BMI training [74]. Such metrics have previously been proposed in the
literature and largely pertain to the different ways to measure the separability of the distributions of the different
mental classes, such as r 2 [37], Fisher score [20], or Kullback-Leibler divergence [42]. The metrics inspired by basic
neuroscience research that effectively describe the same
brain phenomena giving rise to the features used for control can also serve the same purpose adequately [38], [43],
[48], [55]; for instance, a demonstrated increase of eventrelated desynchronization/synchronization in an SMR BMI
based on PSD features.
Open Issues in BMI Mutual Learning
Thus far, only a handful of studies with able-bodied users
[43], [48] and end users [18], [20], [49], [55], [57] adequately
comply with what we have identified as essential prerequisites for a reliable evaluation of subject-learning effects during BMI training: longitudinal monitoring, evaluation at the
BCI feature level, and, ideally, the inclusion of end-user
Ju ly 2020

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE	

15
IEEE Systems, Man and Cybernetics Magazine - July 2020

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - July 2020
