Systems, Man & Cybernetics - July 2017 - 19
soft one; hence, some weights will still be negative.
However, the number and magnitude of these negative
weights are reduced:
2
sl
sl+1
l=1 i=1 j=1
a
a 1 C (w (ilj), l) + 22 w (ilj)
0
2
wi j 1 0
,
wi j $ 0
(9)
where a 1 and a 2 are the nonnegativity-constraint weight
penalty for the L 1 and L 2 terms, respectively.
Although (9) is capable of enforcing nonnegativity in
AEs, its L 1 norm term is nondifferentiable at the origin,
and this can lead to numerical instability during simulations. To circumvent this drawback, smoothing functions
that approximate the L 1 norm are used [13]. One of the
well-known smoothing functions is the quadratic function
given in (10) and depicted in Figure 6. Given any finite
dimensional vector z and positive constant l, the following smoothing function approximates the L 1 norm:
Z || z | |
]
C (z, l) = ][
2
] || z | | + l
2
\ 2l
|| z ||2 l
,
(10)
|| z ||# l
and its gradient is
Z z
] || z | |
]
d z C (z, l) = [
]] z
\l
|| z ||2 l
.
|| z ||# l
(11)
Quadratic, χ
Quadratic, χ
Quadratic, χ
Quadratic, χ
Absolute
1.5
Γ(z, χ)
Decay (w) = | | | *
2
= 0.4
= 0.7
= 1.0
= 1.5
1
0.5
0
−2
−1
0
z
1
2
Figure 6. Absolute function approximations using
quadratic smoothing functions with the parameters
l = 0.4, 0.7, 1.0, and 1.5. l tunes the approximation
of the L 1 norm.
To demonstrate this concept, a subset of the New
York University Object Recognition Benchmark (NORB)
normalized-uniform data set [35] with class labels fourlegged animals, human figures, and airplanes was
extracted. The full data set consists of 24,300 training
images and 24,300 test images of 50 toys from five generic categories: four-legged animals, human figures, airplanes, trucks, and cars. The training and testing sets
comprise five instances of each category. Each image
consists of two channels, each of size 96 × 96 pixels.
(b)
(c)
(a)
(d)
Figure 7. (a) Some sample images from the NORB data set. (b) The weights of the softmax layer are plotted as
a diagram. Each row of the plot corresponds to each output neuron, and each column corresponds to every
hidden neuron. The magnitude of the weight corresponds to the area of each square: white indicates a positive
sign and black indicates a negative one. (c) The RFs learned from the reduced NORB data set using an L1/
L2-NCSAE. The intensity of each pixel is proportional to the magnitude of the weight connected to that pixel
in the input image, with the value zero corresponding to gray. The biases are not shown. (d) The activations of
the hidden neurons for the NORB objects presented in (a) are depicted on the bar charts. Each row shows the
activations of each hidden neuron for five color-coded examples of the same object.
Ju ly 2017
IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE
19
Table of Contents for the Digital Edition of Systems, Man & Cybernetics - July 2017
Systems, Man & Cybernetics - July 2017 - Cover1
Systems, Man & Cybernetics - July 2017 - Cover2
Systems, Man & Cybernetics - July 2017 - 1
Systems, Man & Cybernetics - July 2017 - 2
Systems, Man & Cybernetics - July 2017 - 3
Systems, Man & Cybernetics - July 2017 - 4
Systems, Man & Cybernetics - July 2017 - 5
Systems, Man & Cybernetics - July 2017 - 6
Systems, Man & Cybernetics - July 2017 - 7
Systems, Man & Cybernetics - July 2017 - 8
Systems, Man & Cybernetics - July 2017 - 9
Systems, Man & Cybernetics - July 2017 - 10
Systems, Man & Cybernetics - July 2017 - 11
Systems, Man & Cybernetics - July 2017 - 12
Systems, Man & Cybernetics - July 2017 - 13
Systems, Man & Cybernetics - July 2017 - 14
Systems, Man & Cybernetics - July 2017 - 15
Systems, Man & Cybernetics - July 2017 - 16
Systems, Man & Cybernetics - July 2017 - 17
Systems, Man & Cybernetics - July 2017 - 18
Systems, Man & Cybernetics - July 2017 - 19
Systems, Man & Cybernetics - July 2017 - 20
Systems, Man & Cybernetics - July 2017 - 21
Systems, Man & Cybernetics - July 2017 - 22
Systems, Man & Cybernetics - July 2017 - 23
Systems, Man & Cybernetics - July 2017 - 24
Systems, Man & Cybernetics - July 2017 - 25
Systems, Man & Cybernetics - July 2017 - 26
Systems, Man & Cybernetics - July 2017 - 27
Systems, Man & Cybernetics - July 2017 - 28
Systems, Man & Cybernetics - July 2017 - 29
Systems, Man & Cybernetics - July 2017 - 30
Systems, Man & Cybernetics - July 2017 - 31
Systems, Man & Cybernetics - July 2017 - 32
Systems, Man & Cybernetics - July 2017 - 33
Systems, Man & Cybernetics - July 2017 - 34
Systems, Man & Cybernetics - July 2017 - 35
Systems, Man & Cybernetics - July 2017 - 36
Systems, Man & Cybernetics - July 2017 - 37
Systems, Man & Cybernetics - July 2017 - 38
Systems, Man & Cybernetics - July 2017 - 39
Systems, Man & Cybernetics - July 2017 - 40
Systems, Man & Cybernetics - July 2017 - 41
Systems, Man & Cybernetics - July 2017 - 42
Systems, Man & Cybernetics - July 2017 - 43
Systems, Man & Cybernetics - July 2017 - 44
Systems, Man & Cybernetics - July 2017 - 45
Systems, Man & Cybernetics - July 2017 - 46
Systems, Man & Cybernetics - July 2017 - 47
Systems, Man & Cybernetics - July 2017 - 48
Systems, Man & Cybernetics - July 2017 - Cover3
Systems, Man & Cybernetics - July 2017 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/smc_202310
https://www.nxtbook.com/nxtbooks/ieee/smc_202307
https://www.nxtbook.com/nxtbooks/ieee/smc_202304
https://www.nxtbook.com/nxtbooks/ieee/smc_202301
https://www.nxtbook.com/nxtbooks/ieee/smc_202210
https://www.nxtbook.com/nxtbooks/ieee/smc_202207
https://www.nxtbook.com/nxtbooks/ieee/smc_202204
https://www.nxtbook.com/nxtbooks/ieee/smc_202201
https://www.nxtbook.com/nxtbooks/ieee/smc_202110
https://www.nxtbook.com/nxtbooks/ieee/smc_202107
https://www.nxtbook.com/nxtbooks/ieee/smc_202104
https://www.nxtbook.com/nxtbooks/ieee/smc_202101
https://www.nxtbook.com/nxtbooks/ieee/smc_202010
https://www.nxtbook.com/nxtbooks/ieee/smc_202007
https://www.nxtbook.com/nxtbooks/ieee/smc_202004
https://www.nxtbook.com/nxtbooks/ieee/smc_202001
https://www.nxtbook.com/nxtbooks/ieee/smc_201910
https://www.nxtbook.com/nxtbooks/ieee/smc_201907
https://www.nxtbook.com/nxtbooks/ieee/smc_201904
https://www.nxtbook.com/nxtbooks/ieee/smc_201901
https://www.nxtbook.com/nxtbooks/ieee/smc_201810
https://www.nxtbook.com/nxtbooks/ieee/smc_201807
https://www.nxtbook.com/nxtbooks/ieee/smc_201804
https://www.nxtbook.com/nxtbooks/ieee/smc_201801
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1017
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0717
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0417
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0117
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1016
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0716
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0416
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0116
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_1015
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0715
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0415
https://www.nxtbook.com/nxtbooks/ieee/systems_man_cybernetics_0115
https://www.nxtbookmedia.com