IEEE Systems, Man and Cybernetics Magazine - April 2020 - 55

from Algorithm 4 here is that, previously, we used a criterion that
We were hoping
sought a compromise between the
to pull a rabbit
1-NN error and number of prototypes in the form of a weighted
out of the hat; that
sum. Here, we specify a limit on the
Discussion
is,
identify
niches
number of prototypes. The GA
What is trivial is trivial: When there
turned out to be the best among the
is noise in the data, all points are
that had not been
competitors, which were chosen
higher up, indicating greater error.
explored and propose
from the most successful protoWithout noise, the editing methtype-selection methods [18].
ods (RNN and Hart) were good,
alternative versions
It may be a fluke, but our experiand if we hadn't chosen serendipof the prototype
ment with the GA (and George)
itous parameters of our GA, these
classifier.
showed that this strategy can hanmethods would have been on the
dle noise. However, in both experPareto front. When there is noise,
iments, the generalization error
however, the condensing methods
for M = 200 prototypes increases
learn that noise to perfection, and
the generalization error shoots up (top-right corner of
(possibly due to overfitting), leaving the last point on the
Figure 4). The editing competitors (Wilson and RNGE)
GA line graph out of the Pareto front for the clean data. For
are unfazed by noise. They consistently return good but
the noisy George, the GA with 100 prototypes is marginally
large reference sets. They filter the type of random noise
worse than the Wilson + Hart, another classical hybrid proquite well, and the RNGE found its place in the Pareto
totype-selection method. The random search (MC1) did not
front for the noisy George, beating Wilson by a whisker.
work well here, nor did the RMHC. The likely reason is that
The clear winners are the hybrid methods, a fact that
the class configuration was chosen deliberately to be chalechoes the findings of other authors. We don't need to
lenging, unlike many experimental studies, where the
explicitly enforce the strategy (keep the noise or clean
classes are sampled as Gaussians.
Yes, we evaluate our criterion on the training data.
the noise) within the method; c- riterion-driven methods
This
is what we have been using all the way here (confare a lot better.
densing methods don't have a choice, since they are
What happened with Jim's and my MC1 and GA? In our
meant to guarantee zero resubstitution error). The scat1998 paper [20], we found that random, criterion-driven
terplots, however, show the error on the full data,
methods, such as the MC1 and GA, were simple and effecwhich consists of the 1,000 sampled points (0.33%), and
tive, something that was also mentioned as a surprising
the remaining 299,679 points (99.67%). Don't get me
observation by Skalak [22] in relation to the RMHC. In a
started on the limitations of this example/illustration;
later paper [21], however, we could not confirm this result.
the list is as long as this magazine has pages. But the
My implementation (I will blame that) kicked the GA
moral of the story is that if we need a very small subset
toward the bottom of the league table. The difference
of the data with an acceptable error rate, we may have
to resort to those random, criterion-driven approaches
that seem to offer a good compromise between the car0.26
6
dinality of the reference set and the 1-NN error rate.
RMHC (10)
Long live random search!
0.24
4
RNN
MC1
(10)
Where next? Instance selection from big, semi-super0.22
2
Hart
vised,
streaming, nonstationary, and non-independent and
0.2
2
identically distributed data. Instance selection could be
0.18
8
1-NN
invaluable in that area if we find a smart and successful
GA (10)
0.16
6
RMHC (200)
way of addressing these challenges.
1-NN Error Rate

method on the Pareto front, we show
the cute little portrait of George (the
classification regions, leaving the
background white).

MC1 (200)
GA (197)

0.14
4
0.12
2
0.1
1

Wilson

Wilson and Hart

0.08
8

RNGE

0.06
6
2

2.5

3 3.5 4 4.5 5 5.5 6 6.5
log (Number of Retained Prototypes)

7

Figure 4. The scatterplot of the results for the noisy

George data. The blue line represents the Pareto front.

	

Conclusion
Guess what? That was the conclusion. Back in 1997,
when Jim and I were writing papers together, I would go
to him with a draft, and he would invariably return a
comment: "What kind of conclusion is this? You have run
out of steam, Lucy." And then he would write the conclusion himself. I wish that, one day, I could match Jim's
astute, eloquent, and endlessly entertaining writing. A
girl can dream....
Ap ri l 2020

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE	

55



IEEE Systems, Man and Cybernetics Magazine - April 2020

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - April 2020

Contents
IEEE Systems, Man and Cybernetics Magazine - April 2020 - Cover1
IEEE Systems, Man and Cybernetics Magazine - April 2020 - Cover2
IEEE Systems, Man and Cybernetics Magazine - April 2020 - Contents
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 2
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 3
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 4
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 5
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 6
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 7
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 8
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 9
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 10
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 11
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 12
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 13
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 14
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 15
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 16
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 17
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 18
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 19
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 20
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 21
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 22
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 23
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 24
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 25
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 26
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 27
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 28
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 29
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 30
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 31
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 32
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 33
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 34
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 35
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 36
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 37
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 38
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 39
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 40
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 41
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 42
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 43
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 44
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 45
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 46
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 47
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 48
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 49
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 50
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 51
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 52
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 53
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 54
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 55
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 56
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 57
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 58
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 59
IEEE Systems, Man and Cybernetics Magazine - April 2020 - 60
IEEE Systems, Man and Cybernetics Magazine - April 2020 - Cover3
IEEE Systems, Man and Cybernetics Magazine - April 2020 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/smc_202010
https://www.nxtbook.com/nxtbooks/ieee/smc_202007
https://www.nxtbook.com/nxtbooks/ieee/smc_202004
https://www.nxtbook.com/nxtbooks/ieee/smc_202001
https://www.nxtbook.com/nxtbooks/ieee/smc_201910
https://www.nxtbook.com/nxtbooks/ieee/smc_201907
https://www.nxtbook.com/nxtbooks/ieee/smc_201904
https://www.nxtbook.com/nxtbooks/ieee/smc_201901
https://www.nxtbook.com/nxtbooks/ieee/smc_201810
https://www.nxtbook.com/nxtbooks/ieee/smc_201807
https://www.nxtbook.com/nxtbooks/ieee/smc_201804
https://www.nxtbook.com/nxtbooks/ieee/smc_201801
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1017
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0717
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0417
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0117
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1016
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0716
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0416
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0116
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1015
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0715
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0415
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0115
https://www.nxtbookmedia.com