IEEE Solid-States Circuits Magazine - Fall 2021 - 68

Using a total of four MAC units to perform
32 MAC operations in eight clock cycles indicates
that the design meets the maximum achievable
performance with the available resources.
the operation between an incoming
input vector Xl
rameters F and Dl
vector Yl
F
YW
ll
j
[] /
ii ijDN
ijDN
=+ mod
=
-
1
l
+
#Xllmod
with elements calculated as
[,() ]
[( )],
with the lth layer that
has a weight matrix Wl with hyperparesults
in an output
(9)
which inherently skips the zero values
in Wl
ues that are dilated Dl
for () ,XX which correWWT
Ll
=
P =
1
and of (( ))
l
O logNF NCF
sponds to a cascade of L homogeneous
weighted CSC layers with connectivity
C and fan-out F operating on input vector
X.
For every CSC weight matrix ,Wl
define a compressed format t !
W ,l
following rearrangement:
[, ][ ,( )].mod
t
WW N (10)
l llij =+
ii jD
As a consequence of this, (9) can be
altered as
YX N
llW
[] / t [, ][() ],mod
iiji jDl
=+
=
j
F
-
1
l
(11)
to which we refer as a cyclic dilated
matrix-vector multiplication for a
CSC layer. For FN= and D 1= and
given a new rearrangement of a
dense Wl
into a compressed format
W ,l
t
(11) corresponds to (8), revealing
a novel approach to computation in
standard matrix-vector multiplications
using cyclic dilated matrix-vector
multiplication.
Bottom-Up Training
The DNN model with FC layers is
trained first, and a reference ac68
FALL
2021
we
#
WI ,Rl
NF
which is an N-by-F matrix that contains
only NF nonzero entries of
with the
by taking only the nonzero valelements
apart.
Thus, the computation is of ()O NF for
one single WXl
curacy FCm is obtained. Then, the
FC is replaced with an adjusted CSC
architecture, starting from the most
compressed CSC architecture and
directed toward compression reduction.
With no strict definition, we
consider that the most compressed
CSC architecture is given by small
values for F (e.g., two, three, and
four) if adopting a CSCI architecture
and by C 1=
if adopting a CSCII architecture.
The experiments begin
by targeting the bulky layers of a
DNN. In each experiment, if the accuracy
mCS
C is e less than mFC , the procedure
is terminated, and the CSC
model is accepted. If no CSC layer
replacement satisfies the accuracy
loss, the original layer is restored.
The criterion e is chosen to be 2% in
all experiments of this article.
After training the CSC layers embedded
in the DNN, every weight
matrix Wi
from layer i has nonzero
weights located at corresponding
nonzero elements of its adjacency
matrix
A .i
The weighted CSC layers,
which have substituted one FC
layer, transform the Input to the Output
linearly, given linear activation
for support layers, and can be composed
into an equivalent FC layer with
WW .
T =P =
L
i
-
1
i
It is axiomatic that an
arbitrary FC layer with a weight matrix
WT
cannot be losslessly decomposed
into CSC layers that have fewer synapses
in total. However, since training
does not usually produce a concrete
solution for
W ,T
we use the backpropagation
algorithm to learn good
weights for the CSC layers.
Training Experiments
As depicted in Figure 2, we replace
the FC layers of LeNet-300-100 and
AlexNet with CSCI and CSCII layers
and fine-tune them in 50 epochs. For
AlexNet, the pretrained weights
IEEE SOLID-STATE CIRCUITS MAGAZINE
CSC for Convolution Layers
CSC architectures can be used for convolution
layers as well. Similar to the
scalar values that represent weighted
synapses and I/O nodes of an FC layer,
a traditional 2D convolution layer
can be viewed as an FC graph, as in
Figure 1(a), in which every synapse
and I/O node represent a 2D kernel
and channel, respectively. Thus, every
output node is a 2D channel that
results from summing over all of its
connecting kernels operating on their
connected input channels.
With that perspective, each of
the CSC graphs in Figure 1(c)-(e)
can accommodate a combination of
33# and 11# (or
31 ,#
1 3 ,# and
11 )# kernels to approximate a traditional
2D convolution with 33#
kernels. In [5], CSC architectures
are used to compress convolution
layers of an AlexNet, resulting in
a model compressed by seven and
three times and in size and computation,
respectively.
Hardware
Accelerator Architecture
Similar to GPUs that are engineered
to accelerate matrix operations,
we design a hardware, referred to
as the CSC accelerator processor
(CSC-AP), to efficiently implement
(11), which performs a cyclic dilated
matrix-vector multiplication, i.e.,
obtained from the baseline model are
frozen for the convolution layers, and
only the dense layers are retrained.
Tables 1 and 2 show the compression
and accuracy of the selected
CSC configurations for the two DNNs
in this work and compare them with
related nonstructural pruning methods
from the literature. It is noteworthy
that, in all nonstructural pruning
methods, there exists another implicit
memory space for storing nonzero
weight indexes. Thus, if every nonzero
weight requires an index with the
same number of bits, then the actual
compression in nonstructural pruning
methods is approximately half the reported
rate.
IEEE Solid-States Circuits Magazine - Fall 2021

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2021
