IEEE Solid-States Circuits Magazine - Fall 2021 - 65

Despite the fact that CuSPARSE
can implement the latter problem
6.8 times faster than the CuBLAS, its
advantage is gained by the existence
of 20 times fewer operations, yet its
degraded performance is caused by
the decompressing algorithm applied
to the matrix compressed by the CSR
format. In structured sparsity, on the
other hand, indexing nonzero values
is eliminated, and the implementation
can be handled as high performance
as dense matrices.
Low-Bit-Width Neural Networks
When quantization and pruning methods
are used in tandem, unlike the
DNN weights and activations, the extra
indexing memory imposed by the
pruning method is not quantizable,
and, therefore, the extra indexing
memory might be intolerable in resource-bound
hardware that employs
low bit width, such as binary-/ternaryweight
neural networks. Pruning such
neural networks can result in a larger
model size as compared to their uncompressed
models.
For instance, if a ternary-weight
16,384-by-16,384 matrix with 90%
zero values is compressed with a coordinate
format, it requires 97 MB of
storage (dedicating 1 and 28 b to the
nonzero value and its index, respectively),
whereas its uncompressed
model requires 67 MB (dedicating 2 b
per matrix weight). The second motivation
of devising a CSC architecture
is to develop and employ structurally
compact models that require no indexing
for low-precision neural networks.
Problem Statement
and Formulation
Throughout this work, we use the italic
lowercase letter x to represent generator
polynomials (e.g., ()
px xx1
=+ + 2
),
italic capital letters (e.g., N) for integer
values, and bold uppercase letters for
matrices (e.g., W) and vectors (e.g., X).
We use italic lowercase letters in brackets
for the elements of a matrix or a
vector (e.g., W[i, j]).
Inspired by the butterfly diagrams
of the radix-2 fast Fourier transform
(FFT), we seek SC layers composed
In structured sparsity, on the other hand,
indexing nonzero values is eliminated, and
the implementation can be handled as high
performance as dense matrices.
of a unidirectional sequence of layers:
an Input layer, L 1layers
in between
referred to as support layers,
and an Output layer. We denote the
first (Input) and last (Output) layers
by capitalizing their first letters. All
consecutive layers in the graph are
connected to each other via edges
(synapses). We call this graph homogeneous
if the size of every layer is N
nodes and its intermediate connectivity
is such that the fan-out of every
node in every layer excluding the Output
layer as well as the fan-in of every
layer excluding the Input layer is
exactly F. Therefore, the total number
of edges, E, in this graph is equal to
EN .FL=
(1)
For every layer in the graph, we define
an adjacency matrix, A, whose
length and width are equal to the
input and output sizes of the layer,
respectively (N in this statement)
and whose elements A[i, j] indicate
the number of edges that connect
the input node i to the output node
j. Therefore, NF out of N2
elements
of the adjacency matrix of every support
layer in our problem statement
are ones, and the rest are zeros, indicating
a sparsity of /FN for the
support layer. We then define a merit
of connectivity, C, and constrain the
graph to provide C and only C paths
between every pair of nodes chosen
arbitrarily from the graph Input and
Output layers. For this homogeneous
graph, one can show
FN .CL
=
Proof
Starting from the first layer, every
neuron from the input layer has F possibilities
in every forward hop through
L layers to reach to the output layer
nodes. Thus, NFL
(2)
Also, between every Input/Output pair
exist C paths; therefore, in total there
are CN2
paths. Thus, FN .CL
=
The objective of homogeneity is to
provide a basis where every arbitrary
node in the graph is equally exploited,
every arbitrary pair of nodes from the
Input and Output layers are equally
connected, the flux through the support
layers is fairly equal, and the rank
of the transforming matrix has the
potential to remain full. This foundation
is proposed as an overlay and replacement
for FC layers and defined
such that an FC layer of size N by N is
concluded as a special case in which
FN ,=
L ,1= and C .1= Combining (1)
and (2), a logarithmic relationship between
the number of edges and size of
the layers is inferred as
EN gFNCFlo (),
=
(3)
where E is the number of edges that
corresponds to the number of nonzero
elements of the L layers and
governs the number of multiply-accumulate
(MAC) operations in the
graph. As an example, given F 2=
and
=
=2
2 ,
C ,1= (2) and (3) inferLN2 log
and EN log N which is the case
in radix-2 butterfly diagrams. Figure
1(a)-(e) depicts an FC layer, the
problem statement, and a heterogeneous
and two homogeneous SC
graphs, respectively.
I/O Adjustment
To replace an FC of Input size NI
and Output size NO
with an SC that
has a different value for N, we tile
and truncate only the Input and
Output layers of the SC to match
them to those of the FC layer. To
adjust the Input layer to an input
vector of size
N ,I
paths exist in total.
we remove rows
from the bottom of the adjacency
matrix if NNI
1 (truncation), or
IEEE SOLID-STATE CIRCUITS MAGAZINE
FALL 2021
65

IEEE Solid-States Circuits Magazine - Fall 2021

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2021

Contents
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover1
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover2
IEEE Solid-States Circuits Magazine - Fall 2021 - Contents
IEEE Solid-States Circuits Magazine - Fall 2021 - 2
IEEE Solid-States Circuits Magazine - Fall 2021 - 3
IEEE Solid-States Circuits Magazine - Fall 2021 - 4
IEEE Solid-States Circuits Magazine - Fall 2021 - 5
IEEE Solid-States Circuits Magazine - Fall 2021 - 6
IEEE Solid-States Circuits Magazine - Fall 2021 - 7
IEEE Solid-States Circuits Magazine - Fall 2021 - 8
IEEE Solid-States Circuits Magazine - Fall 2021 - 9
IEEE Solid-States Circuits Magazine - Fall 2021 - 10
IEEE Solid-States Circuits Magazine - Fall 2021 - 11
IEEE Solid-States Circuits Magazine - Fall 2021 - 12
IEEE Solid-States Circuits Magazine - Fall 2021 - 13
IEEE Solid-States Circuits Magazine - Fall 2021 - 14
IEEE Solid-States Circuits Magazine - Fall 2021 - 15
IEEE Solid-States Circuits Magazine - Fall 2021 - 16
IEEE Solid-States Circuits Magazine - Fall 2021 - 17
IEEE Solid-States Circuits Magazine - Fall 2021 - 18
IEEE Solid-States Circuits Magazine - Fall 2021 - 19
IEEE Solid-States Circuits Magazine - Fall 2021 - 20
IEEE Solid-States Circuits Magazine - Fall 2021 - 21
IEEE Solid-States Circuits Magazine - Fall 2021 - 22
IEEE Solid-States Circuits Magazine - Fall 2021 - 23
IEEE Solid-States Circuits Magazine - Fall 2021 - 24
IEEE Solid-States Circuits Magazine - Fall 2021 - 25
IEEE Solid-States Circuits Magazine - Fall 2021 - 26
IEEE Solid-States Circuits Magazine - Fall 2021 - 27
IEEE Solid-States Circuits Magazine - Fall 2021 - 28
IEEE Solid-States Circuits Magazine - Fall 2021 - 29
IEEE Solid-States Circuits Magazine - Fall 2021 - 30
IEEE Solid-States Circuits Magazine - Fall 2021 - 31
IEEE Solid-States Circuits Magazine - Fall 2021 - 32
IEEE Solid-States Circuits Magazine - Fall 2021 - 33
IEEE Solid-States Circuits Magazine - Fall 2021 - 34
IEEE Solid-States Circuits Magazine - Fall 2021 - 35
IEEE Solid-States Circuits Magazine - Fall 2021 - 36
IEEE Solid-States Circuits Magazine - Fall 2021 - 37
IEEE Solid-States Circuits Magazine - Fall 2021 - 38
IEEE Solid-States Circuits Magazine - Fall 2021 - 39
IEEE Solid-States Circuits Magazine - Fall 2021 - 40
IEEE Solid-States Circuits Magazine - Fall 2021 - 41
IEEE Solid-States Circuits Magazine - Fall 2021 - 42
IEEE Solid-States Circuits Magazine - Fall 2021 - 43
IEEE Solid-States Circuits Magazine - Fall 2021 - 44
IEEE Solid-States Circuits Magazine - Fall 2021 - 45
IEEE Solid-States Circuits Magazine - Fall 2021 - 46
IEEE Solid-States Circuits Magazine - Fall 2021 - 47
IEEE Solid-States Circuits Magazine - Fall 2021 - 48
IEEE Solid-States Circuits Magazine - Fall 2021 - 49
IEEE Solid-States Circuits Magazine - Fall 2021 - 50
IEEE Solid-States Circuits Magazine - Fall 2021 - 51
IEEE Solid-States Circuits Magazine - Fall 2021 - 52
IEEE Solid-States Circuits Magazine - Fall 2021 - 53
IEEE Solid-States Circuits Magazine - Fall 2021 - 54
IEEE Solid-States Circuits Magazine - Fall 2021 - 55
IEEE Solid-States Circuits Magazine - Fall 2021 - 56
IEEE Solid-States Circuits Magazine - Fall 2021 - 57
IEEE Solid-States Circuits Magazine - Fall 2021 - 58
IEEE Solid-States Circuits Magazine - Fall 2021 - 59
IEEE Solid-States Circuits Magazine - Fall 2021 - 60
IEEE Solid-States Circuits Magazine - Fall 2021 - 61
IEEE Solid-States Circuits Magazine - Fall 2021 - 62
IEEE Solid-States Circuits Magazine - Fall 2021 - 63
IEEE Solid-States Circuits Magazine - Fall 2021 - 64
IEEE Solid-States Circuits Magazine - Fall 2021 - 65
IEEE Solid-States Circuits Magazine - Fall 2021 - 66
IEEE Solid-States Circuits Magazine - Fall 2021 - 67
IEEE Solid-States Circuits Magazine - Fall 2021 - 68
IEEE Solid-States Circuits Magazine - Fall 2021 - 69
IEEE Solid-States Circuits Magazine - Fall 2021 - 70
IEEE Solid-States Circuits Magazine - Fall 2021 - 71
IEEE Solid-States Circuits Magazine - Fall 2021 - 72
IEEE Solid-States Circuits Magazine - Fall 2021 - 73
IEEE Solid-States Circuits Magazine - Fall 2021 - 74
IEEE Solid-States Circuits Magazine - Fall 2021 - 75
IEEE Solid-States Circuits Magazine - Fall 2021 - 76
IEEE Solid-States Circuits Magazine - Fall 2021 - 77
IEEE Solid-States Circuits Magazine - Fall 2021 - 78
IEEE Solid-States Circuits Magazine - Fall 2021 - 79
IEEE Solid-States Circuits Magazine - Fall 2021 - 80
IEEE Solid-States Circuits Magazine - Fall 2021 - 81
IEEE Solid-States Circuits Magazine - Fall 2021 - 82
IEEE Solid-States Circuits Magazine - Fall 2021 - 83
IEEE Solid-States Circuits Magazine - Fall 2021 - 84
IEEE Solid-States Circuits Magazine - Fall 2021 - 85
IEEE Solid-States Circuits Magazine - Fall 2021 - 86
IEEE Solid-States Circuits Magazine - Fall 2021 - 87
IEEE Solid-States Circuits Magazine - Fall 2021 - 88
IEEE Solid-States Circuits Magazine - Fall 2021 - 89
IEEE Solid-States Circuits Magazine - Fall 2021 - 90
IEEE Solid-States Circuits Magazine - Fall 2021 - 91
IEEE Solid-States Circuits Magazine - Fall 2021 - 92
IEEE Solid-States Circuits Magazine - Fall 2021 - 93
IEEE Solid-States Circuits Magazine - Fall 2021 - 94
IEEE Solid-States Circuits Magazine - Fall 2021 - 95
IEEE Solid-States Circuits Magazine - Fall 2021 - 96
IEEE Solid-States Circuits Magazine - Fall 2021 - 97
IEEE Solid-States Circuits Magazine - Fall 2021 - 98
IEEE Solid-States Circuits Magazine - Fall 2021 - 99
IEEE Solid-States Circuits Magazine - Fall 2021 - 100
IEEE Solid-States Circuits Magazine - Fall 2021 - 101
IEEE Solid-States Circuits Magazine - Fall 2021 - 102
IEEE Solid-States Circuits Magazine - Fall 2021 - 103
IEEE Solid-States Circuits Magazine - Fall 2021 - 104
IEEE Solid-States Circuits Magazine - Fall 2021 - 105
IEEE Solid-States Circuits Magazine - Fall 2021 - 106
IEEE Solid-States Circuits Magazine - Fall 2021 - 107
IEEE Solid-States Circuits Magazine - Fall 2021 - 108
IEEE Solid-States Circuits Magazine - Fall 2021 - 109
IEEE Solid-States Circuits Magazine - Fall 2021 - 110
IEEE Solid-States Circuits Magazine - Fall 2021 - 111
IEEE Solid-States Circuits Magazine - Fall 2021 - 112
IEEE Solid-States Circuits Magazine - Fall 2021 - 113
IEEE Solid-States Circuits Magazine - Fall 2021 - 114
IEEE Solid-States Circuits Magazine - Fall 2021 - 115
IEEE Solid-States Circuits Magazine - Fall 2021 - 116
IEEE Solid-States Circuits Magazine - Fall 2021 - 117
IEEE Solid-States Circuits Magazine - Fall 2021 - 118
IEEE Solid-States Circuits Magazine - Fall 2021 - 119
IEEE Solid-States Circuits Magazine - Fall 2021 - 120
IEEE Solid-States Circuits Magazine - Fall 2021 - 121
IEEE Solid-States Circuits Magazine - Fall 2021 - 122
IEEE Solid-States Circuits Magazine - Fall 2021 - 123
IEEE Solid-States Circuits Magazine - Fall 2021 - 124
IEEE Solid-States Circuits Magazine - Fall 2021 - 125
IEEE Solid-States Circuits Magazine - Fall 2021 - 126
IEEE Solid-States Circuits Magazine - Fall 2021 - 127
IEEE Solid-States Circuits Magazine - Fall 2021 - 128
IEEE Solid-States Circuits Magazine - Fall 2021 - 129
IEEE Solid-States Circuits Magazine - Fall 2021 - 130
IEEE Solid-States Circuits Magazine - Fall 2021 - 131
IEEE Solid-States Circuits Magazine - Fall 2021 - 132
IEEE Solid-States Circuits Magazine - Fall 2021 - 133
IEEE Solid-States Circuits Magazine - Fall 2021 - 134
IEEE Solid-States Circuits Magazine - Fall 2021 - 135
IEEE Solid-States Circuits Magazine - Fall 2021 - 136
IEEE Solid-States Circuits Magazine - Fall 2021 - 137
IEEE Solid-States Circuits Magazine - Fall 2021 - 138
IEEE Solid-States Circuits Magazine - Fall 2021 - 139
IEEE Solid-States Circuits Magazine - Fall 2021 - 140
IEEE Solid-States Circuits Magazine - Fall 2021 - 141
IEEE Solid-States Circuits Magazine - Fall 2021 - 142
IEEE Solid-States Circuits Magazine - Fall 2021 - 143
IEEE Solid-States Circuits Magazine - Fall 2021 - 144
IEEE Solid-States Circuits Magazine - Fall 2021 - 145
IEEE Solid-States Circuits Magazine - Fall 2021 - 146
IEEE Solid-States Circuits Magazine - Fall 2021 - 147
IEEE Solid-States Circuits Magazine - Fall 2021 - 148
IEEE Solid-States Circuits Magazine - Fall 2021 - 149
IEEE Solid-States Circuits Magazine - Fall 2021 - 150
IEEE Solid-States Circuits Magazine - Fall 2021 - 151
IEEE Solid-States Circuits Magazine - Fall 2021 - 152
IEEE Solid-States Circuits Magazine - Fall 2021 - 153
IEEE Solid-States Circuits Magazine - Fall 2021 - 154
IEEE Solid-States Circuits Magazine - Fall 2021 - 155
IEEE Solid-States Circuits Magazine - Fall 2021 - 156
IEEE Solid-States Circuits Magazine - Fall 2021 - 157
IEEE Solid-States Circuits Magazine - Fall 2021 - 158
IEEE Solid-States Circuits Magazine - Fall 2021 - 159
IEEE Solid-States Circuits Magazine - Fall 2021 - 160
IEEE Solid-States Circuits Magazine - Fall 2021 - 161
IEEE Solid-States Circuits Magazine - Fall 2021 - 162
IEEE Solid-States Circuits Magazine - Fall 2021 - 163
IEEE Solid-States Circuits Magazine - Fall 2021 - 164
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover3
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2019
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2019
https://www.nxtbook.com/nxtbooks/ieee/mssc_2019summer
https://www.nxtbook.com/nxtbooks/ieee/mssc_2019winter
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018fall
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018summer
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018spring
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018winter
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2014
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2014
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2014
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2014
https://www.nxtbookmedia.com