Systems, Man & Cybernetics - July 2015 - 22

To summarize, CNNs have the following apparent
drawbacks and limitations:
◆ BP learning speed and generalization performance for
limited training data sets
◆ tweaks and tricks should be adopted to define the
architecture parameters.
Extreme Learning Machines
ANNs and CNNs have been widely explored as architectures to handle complex and challenging tasks. While new
learning techniques have been proposed by the community, almost all of them are derived from the BP algorithm.
Hence, such networks lack faster learning procedures. It
is, therefore, not surprising to see that it may take hours
[40], days [4], and even longer to train a single ANN/CNN
using the traditional BP-based methods.
Biological-Inspired Learning
Extreme learning machines (ELMs) [41] were inspired by
biological learning and proposed to overcome the issues
faced by the BP-based learning algorithms. ELMs were
devised following the idea that some parts of human brain
systems have random neurons with parameters that are
independent of their environment. All of the hidden nodes
in an ELM are independent of the training data as well as
each other (Figure 3). Hidden nodes need not be tuned, and
the input-to-hidden weights can be randomly generated
before seeing the training data. The solution to learn the
optimal hidden-to-output weights (to obtain the correct
input-output mappings) is the Moore-Penrose generalized
inverse [42].
ELMs: Advantages and Shortcomings
ELMs are extremely fast learning algorithms for SLFNs,
and they have the following important properties.

Input Layer

Hidden Layer

Output Layer

Forward Propagation
b1,
1
h1
Random
ts
h
ig
e
W
h2

yt 1
yt 2

x2
b 1,

hL - 2

yt D

hL - 1

Figure 3. The architecture of an ELM. Unlike previously

seen neural architectures, there is no BP in the
learning process. With the ELM, the input-to-hidden
weights are randomly picked. Only the set of hiddento-output connection weights b are learned.

IEEE Systems, Man, & Cybernetics Magazine July 2015

◆ Minimum training error: The special solution for

learning the hidden-to-output layer connection
weights, derived from exploiting the properties of the
network structure and the Moore-Penrose generalized inverse matrix [42], is one of the least-square
solutions of a linear system.
◆ The smallest norm of weights and best generalization performance: Bartlett's neural network generalization theory [43] for feedforward neural networks
asserts that the smaller the norms of weights are,
the better generalization performance the networks
tend to have. ELMs have this property; indeed, the
solution has the smallest norm among all the leastsquare solutions.
◆ Unique minimum norm least-square solution: The
minimum norm solution for learning the hidden-tooutput weights is unique.
While ELMs have valuable properties, their success
or failure, like ANN and NT architectures, hinges on the
adopted data representation.
The Future of Brain-Inspired Learning
Architectures: Extreme Deep Learning Trees
Considering the strengths and weaknesses of each of the
previously presented architectures, we propose a novel
architecture that borrows their strengths and combines
them into a unique system. We named the architecture the
extreme deep learning tree (EDLT).
EDLT Structure
The EDLT exploits an NT architecture whose in-node
models are composed of an ELM stacked at the output of a
three-layer CNN (i.e., a CNN composed of a convolutional,
an LCN, and a pooling layer). See Figure 4 for the in-node
model architecture.
The Tree: Less Prior Knowledge
One of the main current issues of ANNs is correctly determining the number and the arrangement of hidden layers and neurons before the training starts. A common
approach is to train many different networks (each with a
different structure) and then to adopt the configuration of
the one that has the minimum validation error. However,
even if the optimal configuration for a particular task can
be found, it is not certain that the chosen configuration
will be the best one. The possible configurations of a
single network with only one hidden layer are infinite (we
can have one layer with infinite hidden neurons). The tree
structure of an NT, together with the adoption of appropriate split nodes, allows us to solve such a problem.
CNN Layers: Random Feature Extraction
We have shown that CNNs are able to learn meaningful
feature extractors only by looking at the data. However,
this process requires a network to perform several forward/backward propagations over the data. In addition

Table of Contents for the Digital Edition of Systems, Man & Cybernetics - July 2015