IEEE Circuits and Systems Magazine - Q2 2018 - 87

Image Size:
28 × 28

Layer 1

Input Size: 28 × 28
Output Size: 26 × 26

Convop1,c1

Convop1,c2

Convop1,c3

Convop1,c4

Layer 2

Input Size: 26 × 26
Output Size: 26 × 26

ReLUop1,c1

ReLUop1,c2

ReLUop1,c3

ReLUop1,c4

Layer 3

Input Size: 26 × 26
Output Size: 24 × 24

Poolop1,c1

Poolop1,c2

Poolop1,c3

Poolop1,c4

Layer 4

Input Size: 24 × 24
Output Size: 22 × 22

Convop2,c1

Convop2,c2

Convop2,c3

Convop2,c4

Layer 5

Input Size: 22 × 22
Output Size: 22 × 22

ReLUop2,c1

ReLUop2,c2

ReLUop2,c3

ReLUop2,c4

Layer 6

Input Size: 22 × 22
Output Size: 20 × 20

Poolop2,c1

Poolop2,c2

Poolop2,c3

Poolop2,c4

Fully
Connected
Figure 15. the architecture of the convolutional network containing two regular convolutional layers each of them are built up by
a convolution, a rectifier and a pooling operation.

were developed which try to incorporate different factors of the implementation (e.g.: price, accuracy, power
consumption, delay etc.) and in case of practical problems one cannot restrict themselves to "traditional"
metrics such as independent measurements of accuracy, energy or delay. In [38] a CNN-friendly structure was
introduced executing a convolutional neural network for
the MNIST problem. MNIST is handwritten digit classification task [39]. A system must analyze and classify
what digit (0-9) is represented by a 28 # 28 pixel black
and white image. There are 60,000 images in the training set, and 10,000 images in the test set. It was shown
that all operations required by convolutional neural
networks: the convolution, ReLU, and pooling steps
are feed-forward (B-type) CNN templates. The feedback
template (A) is not used in any of the feature extracting
operations (i.e., per Eq. 1, all values would simply be 0).
As such, the training of the network can also be done
with a CNN using commonly applied techniques in deep
learning such as backpropagation stochastic gradient
descent. Additionally, all computational kernels are restricted to a CNN friendly size of neighbourhood radius
one, 3 # 3 templates. As it can also be seen from the
Theory of CNNs (in Section II) larger kernels are also
allowed, but practical and hardware implementations
SECOND quartEr 2018

usually reason the application of 3 # 3 kernels. Furthermore, recent work [40] suggests that smaller kernels can
lead to fewer parameters/higher accuracy during training which ensure efficient mapping of the algorithms to
CNN hardware: initial numerical simulation-with 32-bit
precision-results suggest that a CNN-friendly convolutional neural network has a classification accuracy of
97%, which is competitive with other approaches in the
published literature. Such an architecture can be seen
on Fig 15.
We have to note that CNNs are analog in nature which can
result increased efficiency, but we also have to consider
this fact during comparison with digital architectures. It
is not reasonable to expect 32-bit digital precision from
analogue devices and CNN hardwares. As it was investigated in [38] even with just 4-bit precision (i.e., to represent template values for convolution operations) classification accuracy still remains above 96%. The energy/
delay projections along with accuracy were compared to
the TrueNorth chip of IBM (28 nm) and to an Intel i5, 14
nm system with an Nvidia Tesla K80 GPU.
Classification accuracy as well as energy/delay per
classification for other approaches to solving the MNIST
problem are summarized in Table II. TrueNorth (TN) data
(28 nm) is extracted from [41] and [42], while DropConnect
IEEE CIrCuItS aND SyStEmS magazINE

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2018