Systems, Man & Cybernetics - April 2016 - 6

EM method was raised by
equation was discovered (created),
several discussants of [8].
and popularized (spread around),
The interplay between
For example, Hartley opened
by Albert Einstein. But suppose I
AI and CI is, in some
his contribution with
ment ion back propagat ion i n
I felt like the old minstrel
multilayered neural networks.
sense, just an offshoot
who has been singing his
What name leaps to your mind?
of never-ending
song for 18 years and now
This very famous and useful
finds, with considerable sattechnique was created and re arguments about the
isfaction, that his folklore is
ported by Paul Werbos in his Ph.D.
meaning of the root
the theme of an overpowerdissertation [4], but it was, for
ing symphony.
many years, erroneously credited
word intelligence.
Hartley's folk-song analoto David Rumelhart and James
gy is indeed appropriate for
McClelland, who popularized it in
describing the development
their 1986 book [5]. Where does all
of this powerful method. Just as a folk-song typicalthis lead? Suppose I ask the membership of the IEEE who
ly evolves many years before its tune is well recogoriginated the term CI. Many-perhaps most-of them
nized, various EM-type methods or ideas which premight say "Jim Bezdek," but they would be wrong. Just as in
cede Dempster et al., and in fact precede Hartley by
the case of Goertzel's term AGI, I probably made this term
many years, can be found in the literature.
a big hit (spread it around), but I did not write the song.
Although we shall perhaps never be able to find
That piece of the story will come to light soon. But first,
out who really sang the first musical note of the EM
let's return to the music analogy, which was not my
algorithm, we all agree that it was Dempster et al.
invention either!
who brought it into the all-time top 10 of statistics
(see Stigler [9]). They made (at least) two contribuFolk Songs and Smash Hits
tions that popularized the song. First, they gave it
There is a close parallel between the rise of (the term
an informative title identifying the key stanzas-the
for) CI and the common abbreviation for the expectaexpectation step and the maximization step.
tion-maximization (EM) algorithm. I have done a lot of
Second, they demonstrated how it could be sung at
work related to the theory of alternating optimization
many different occasions, some of which had not
(AO), which is the scheme employed by EM when estipreviously been thought to be related to the EM
mating the unknown parameters of a bunch of mixed
algorithm (e.g. viewing latent variables as missing
probability distributions. In due course, I became very
data). Since then, we all have sung or heard it being
interested in trying to track down the history of AO, and
sung many times, sometimes with abusive or even
my inquiries into this topic became somewhat inseparaunbearable tones. [6]
ble from the history of the EM algorithm. Let me call
My poi nt is that th is t y pe of con fusion-who
this combined history EM/AO. It turns out that this hisinitiated a term versus who spread it around-is quite
tory is, unsurprisingly, pretty cloudy. Several scholars
common in both the arts and in the sciences. There is,
have written quite interesting and rather comprehensive
more or less a direct connection between the music
treatises about the history of various EM/AO algoanalogy I made previously and the popularization of the
rithms. But the history of the term EM itself is pretty
well known, and delightfully rendered by Meng and Van
terms EM and CI. The history lesson offered by Meng
Dyk [6], who wrote (I have changed their references to
and Van Dyk adapts remarkably well to the history
the IEEE format and emphasized the crucial fact about
of evolution of the ter m CI. But CI is not qu ite
this term in italics):
semantically equivalent to EM, because EM refers to
several real AO algorithms, whereas CI is simply a
broad-brush term that is used to describe-what? Well,
How EM Became a Monster Hit
that's the point of this treatise, isn't it?
Who First Developed the EM Algorithm? With the
ever-growing popularity of the EM algorithm, espeComputational Intelligence Begins
cially with its various deterministic and stochastic
The appearance of the term CI in published form goes
extensions (e.g. the data augmentation algorithm of
back to at least 1983, for that is when the International
Tanner and Wong [7]), those of us who do research
Journal of Computational Intelligence (IJCI) was floated
in this area find ourselves being asked more freas the title of a new Canadian journal by its founding ediquently the question who first developed the EM
tors, Nick Cercone and Gordon McCalla. Nick and Gordon
algorithm? Although it is easy for us to direct the
both responded to my request for some information on
inquirer to Dempster et al. [8], where the term EM
their use of the term. Here is what each of them wrote to
appeared for the first time, the question is really not
me in e-mail communications.
easy to answer. In fact, the issue of the origin of the
6

IEEE SyStEmS, man, & CybErnEtICS magazInE A pri l 2016



Table of Contents for the Digital Edition of Systems, Man & Cybernetics - April 2016

Systems, Man & Cybernetics - April 2016 - Cover1
Systems, Man & Cybernetics - April 2016 - Cover2
Systems, Man & Cybernetics - April 2016 - 1
Systems, Man & Cybernetics - April 2016 - 2
Systems, Man & Cybernetics - April 2016 - 3
Systems, Man & Cybernetics - April 2016 - 4
Systems, Man & Cybernetics - April 2016 - 5
Systems, Man & Cybernetics - April 2016 - 6
Systems, Man & Cybernetics - April 2016 - 7
Systems, Man & Cybernetics - April 2016 - 8
Systems, Man & Cybernetics - April 2016 - 9
Systems, Man & Cybernetics - April 2016 - 10
Systems, Man & Cybernetics - April 2016 - 11
Systems, Man & Cybernetics - April 2016 - 12
Systems, Man & Cybernetics - April 2016 - 13
Systems, Man & Cybernetics - April 2016 - 14
Systems, Man & Cybernetics - April 2016 - 15
Systems, Man & Cybernetics - April 2016 - 16
Systems, Man & Cybernetics - April 2016 - 17
Systems, Man & Cybernetics - April 2016 - 18
Systems, Man & Cybernetics - April 2016 - 19
Systems, Man & Cybernetics - April 2016 - 20
Systems, Man & Cybernetics - April 2016 - 21
Systems, Man & Cybernetics - April 2016 - 22
Systems, Man & Cybernetics - April 2016 - 23
Systems, Man & Cybernetics - April 2016 - 24
Systems, Man & Cybernetics - April 2016 - 25
Systems, Man & Cybernetics - April 2016 - 26
Systems, Man & Cybernetics - April 2016 - 27
Systems, Man & Cybernetics - April 2016 - 28
Systems, Man & Cybernetics - April 2016 - 29
Systems, Man & Cybernetics - April 2016 - 30
Systems, Man & Cybernetics - April 2016 - 31
Systems, Man & Cybernetics - April 2016 - 32
Systems, Man & Cybernetics - April 2016 - 33
Systems, Man & Cybernetics - April 2016 - 34
Systems, Man & Cybernetics - April 2016 - 35
Systems, Man & Cybernetics - April 2016 - 36
Systems, Man & Cybernetics - April 2016 - 37
Systems, Man & Cybernetics - April 2016 - 38
Systems, Man & Cybernetics - April 2016 - 39
Systems, Man & Cybernetics - April 2016 - 40
Systems, Man & Cybernetics - April 2016 - 41
Systems, Man & Cybernetics - April 2016 - 42
Systems, Man & Cybernetics - April 2016 - 43
Systems, Man & Cybernetics - April 2016 - 44
Systems, Man & Cybernetics - April 2016 - 45
Systems, Man & Cybernetics - April 2016 - 46
Systems, Man & Cybernetics - April 2016 - 47
Systems, Man & Cybernetics - April 2016 - 48
Systems, Man & Cybernetics - April 2016 - 49
Systems, Man & Cybernetics - April 2016 - 50
Systems, Man & Cybernetics - April 2016 - 51
Systems, Man & Cybernetics - April 2016 - 52
Systems, Man & Cybernetics - April 2016 - 53
Systems, Man & Cybernetics - April 2016 - 54
Systems, Man & Cybernetics - April 2016 - 55
Systems, Man & Cybernetics - April 2016 - 56
Systems, Man & Cybernetics - April 2016 - Cover3
Systems, Man & Cybernetics - April 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/smc_202110
https://www.nxtbook.com/nxtbooks/ieee/smc_202107
https://www.nxtbook.com/nxtbooks/ieee/smc_202104
https://www.nxtbook.com/nxtbooks/ieee/smc_202101
https://www.nxtbook.com/nxtbooks/ieee/smc_202010
https://www.nxtbook.com/nxtbooks/ieee/smc_202007
https://www.nxtbook.com/nxtbooks/ieee/smc_202004
https://www.nxtbook.com/nxtbooks/ieee/smc_202001
https://www.nxtbook.com/nxtbooks/ieee/smc_201910
https://www.nxtbook.com/nxtbooks/ieee/smc_201907
https://www.nxtbook.com/nxtbooks/ieee/smc_201904
https://www.nxtbook.com/nxtbooks/ieee/smc_201901
https://www.nxtbook.com/nxtbooks/ieee/smc_201810
https://www.nxtbook.com/nxtbooks/ieee/smc_201807
https://www.nxtbook.com/nxtbooks/ieee/smc_201804
https://www.nxtbook.com/nxtbooks/ieee/smc_201801
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1017
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0717
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0417
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0117
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1016
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0716
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0416
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0116
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1015
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0715
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0415
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0115
https://www.nxtbookmedia.com