IEEE Circuits and Systems Magazine - Q4 2019 - 22

anti-spoofing methods like [13], [14], deep learning based
approaches provide improved capability for protecting
automatic speaker verification (ASV) systems from speaker spoofing via synthetic speech [15].
Besides the acoustic signals produced by human vocal systems (e.g. speech and song), another ultimate
acoustic 'language' is that of music. To compose a natural-sounding music, several types of musical content
need to be generated, such as the melody, polyphony,
counterpoint, chords and lead sheet. By a combination
of all these musical components, music, which is wellknown for its emotional effect, can act as a form of expression and preciseness [16]. More details about music
generation can be found in refs. [17], [18].
Machine learning has been the dominating technology in a variety of signal processing applications. In processing raw data from the natural information source,
conventional machine learning approaches are limited
by the complicated requirements of feature extraction
[19]. Deep learning (DL), as a subfield of machine learning, involves a hierarchical computing framework in
which multiple layers of learning algorithms are stacked
in a specific order. With an approximation of nonlinear functions, deep learning algorithms are applied to
achieve abstract representations and learn feature vectors from the original data. We have seen impressive
progress of deep learning over the last decade, to many
engineering and science application areas ranging from
web searches, online content filtering and recommendations systems to computer vision, speech recognition
and other machine intelligence tasks [20].
In Fig. 3 we depict the historical development of the
deep learning algorithms discussed in this paper. The artificial neural network (ANN) [21] originated in the 1940s

Recurrent
Neural
Network

and led to the first wave of artificial intelligence (AI) algorithms with the creation of the single-layer perceptron
(SLP) and the multi-layer perceptron (MLP) [22], [23]. To
build a standard neural network (NN), varying amounts
of neurons are used to yield real-valued activations and,
by adjusting the weights and biases, the NNs can behave as expected in specific tasks like computer vision
and speech recognition [24]. The development of ANNs
stagnated because of the incapability of processing the
exclusive-or circuit and computing devices with low
processing capacity. The next wave of research boomed
in the 1980s when the backpropagation algorithm (BP)
[25] was proposed. As an efficient gradient descent algorithm, BP effectively accelerated the training of ANNs.
However, in the late 1990s, ANNs and the BP algorithm
were largely abandoned by the community. It was generally believed that backpropagation would get trapped in
poor local minima and the average error could not be reduced any more. In addition to that, insufficient labeled
training data can cause the problem of over-fitting [19].
In 2006, which is considered the first year of deep
learning, several breakthroughs regarding new network
architectures and training methods revived the interest in neural networks. In [26], a new layer-wise-greedylearning based method was proposed for training very
deep neural networks (DNN). Hidden layers in a network
are pre-trained one layer at a time using the unsupervised learning approach and this considerably helps
to accelerate subsequent supervised learning through
the backpropagation algorithm [19], [27]. Also in 2006, a
Convolutional Neural Network (CNN) trained by BP set
a new record of 0.39% on the handwriting digits database MNIST [28], which was a significant progress in the
performance since the classical prototype LeNet-5 [29].

Deep Neural
Network
(Pretraining)

Variational
AutoEncoder

Perceptron

1940

1980

Multi-Layer
Perceptron

2000

Restricted
Boltzmann
Machine

Generative
Adversarial
Network

2010

Convolutional
Neural
Network

Deep
Belief
Network

Wasserstein
GAN

Figure 3. A timeline of deep learning algorithms introduced.
22

IEEE CIRCUITS AND SYSTEMS MAGAZINE

FOURTH QUARTER 2019
IEEE Circuits and Systems Magazine - Q4 2019

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2019
