IEEE Solid-States Circuits Magazine - Fall 2021 - 64

Morteza Hosseini, Nitheesh Manjunath, Uttej Kallakuri,
Hamid Mahmoodi, Houman Homayoun, and Tinoosh Mohsenin
Connected Architectures
Cyclic Sparsely
From foundations
to applications
D
log
eep neural networks
(DNNs) are
compute-intensive
nonlinear
mathematical
functions
that employ matrix/tensor
operators at their core to identify
temporal and/or spatial correlations
within input data. Common techniques,
such as pruning, quantization,
and compact model design,
have been proposed by researchers
and extensively utilized by developers
to reduce the computation and
bulky size of DNNs.
Under the category of compact
model design, cyclic sparsely
connected (CSC) architectures [1]
are structurally sparsified graphs
that can be used to effectively compress
DNNs as an alternative to
pruning methods with the advantage
of imposing less of a memory
footprint. CSC architectures have a
memory/computation complexity of
O ()
NN that can be used as an
overlay for a fully connected (FC)
of (),O N2
where N is the number of
input-output (I/O) nodes given an
equally sized FC layer.
Digital Object Identifier 10.1109/MSSC.2021.3111431
Date of current version: 17 November 2021
64
Fully Connected
Cyclic Sparsely Connected
Throughout this article, we focus
on the complexity of FC layers and
how CSC architectures can be used as
a compact overlay for them in DNNs.
In the end, we briefly remark on how
CSC architectures can be used for
compact convolution layers, too.
Motivation and Background
Pruning Versus Structured Sparsity
Despite their effectiveness, pruning
methods result in the irregularity of
the pattern of nonzero weights in the
pruned model of a neural network,
necessitating an additional indexing.
Thus, their compressed model
has more parameters than the sheer
number of nonzero weights, and their
implementation is deteriorated by the
model decompression.
FALL 2021
IEEE SOLID-STATE CIRCUITS MAGAZINE
As an example, our experiments
on the NVIDIA TX2 GPU at a clock
rate of 1.3 GHz indicates that a matrix-vector
multiplication using a matrix
of size 16,384 by 16,384 can be
implemented by the NVIDIA CuBLAS
library, which is a GPU-accelerated
implementation library of the basic
linear algebra subroutines (BLAS),
with a performance of approximately
11.4 giga operations per second
(GOPS), whereas the implementation
of the same matrix with 95% zero values
compressed with the compressed
sparse row (CSR) storage format can be
executed using the CuSPARSE library,
which is a GPU-accelerated implementation
library for sparse matrices, with
a performance of approximately 3.9
GOPS that merely operates over the 5%
nonzero values.
1943-0582/21©2021IEEE

IEEE Solid-States Circuits Magazine - Fall 2021

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2021

Contents
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover1
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover2
IEEE Solid-States Circuits Magazine - Fall 2021 - Contents
IEEE Solid-States Circuits Magazine - Fall 2021 - 2
IEEE Solid-States Circuits Magazine - Fall 2021 - 3
IEEE Solid-States Circuits Magazine - Fall 2021 - 4
IEEE Solid-States Circuits Magazine - Fall 2021 - 5
IEEE Solid-States Circuits Magazine - Fall 2021 - 6
IEEE Solid-States Circuits Magazine - Fall 2021 - 7
IEEE Solid-States Circuits Magazine - Fall 2021 - 8
IEEE Solid-States Circuits Magazine - Fall 2021 - 9
IEEE Solid-States Circuits Magazine - Fall 2021 - 10
IEEE Solid-States Circuits Magazine - Fall 2021 - 11
IEEE Solid-States Circuits Magazine - Fall 2021 - 12
IEEE Solid-States Circuits Magazine - Fall 2021 - 13
IEEE Solid-States Circuits Magazine - Fall 2021 - 14
IEEE Solid-States Circuits Magazine - Fall 2021 - 15
IEEE Solid-States Circuits Magazine - Fall 2021 - 16
IEEE Solid-States Circuits Magazine - Fall 2021 - 17
IEEE Solid-States Circuits Magazine - Fall 2021 - 18
IEEE Solid-States Circuits Magazine - Fall 2021 - 19
IEEE Solid-States Circuits Magazine - Fall 2021 - 20
IEEE Solid-States Circuits Magazine - Fall 2021 - 21
IEEE Solid-States Circuits Magazine - Fall 2021 - 22
IEEE Solid-States Circuits Magazine - Fall 2021 - 23
IEEE Solid-States Circuits Magazine - Fall 2021 - 24
IEEE Solid-States Circuits Magazine - Fall 2021 - 25
IEEE Solid-States Circuits Magazine - Fall 2021 - 26
IEEE Solid-States Circuits Magazine - Fall 2021 - 27
IEEE Solid-States Circuits Magazine - Fall 2021 - 28
IEEE Solid-States Circuits Magazine - Fall 2021 - 29
IEEE Solid-States Circuits Magazine - Fall 2021 - 30
IEEE Solid-States Circuits Magazine - Fall 2021 - 31
IEEE Solid-States Circuits Magazine - Fall 2021 - 32
IEEE Solid-States Circuits Magazine - Fall 2021 - 33
IEEE Solid-States Circuits Magazine - Fall 2021 - 34
IEEE Solid-States Circuits Magazine - Fall 2021 - 35
IEEE Solid-States Circuits Magazine - Fall 2021 - 36
IEEE Solid-States Circuits Magazine - Fall 2021 - 37
IEEE Solid-States Circuits Magazine - Fall 2021 - 38
IEEE Solid-States Circuits Magazine - Fall 2021 - 39
IEEE Solid-States Circuits Magazine - Fall 2021 - 40
IEEE Solid-States Circuits Magazine - Fall 2021 - 41
IEEE Solid-States Circuits Magazine - Fall 2021 - 42
IEEE Solid-States Circuits Magazine - Fall 2021 - 43
IEEE Solid-States Circuits Magazine - Fall 2021 - 44
IEEE Solid-States Circuits Magazine - Fall 2021 - 45
IEEE Solid-States Circuits Magazine - Fall 2021 - 46
IEEE Solid-States Circuits Magazine - Fall 2021 - 47
IEEE Solid-States Circuits Magazine - Fall 2021 - 48
IEEE Solid-States Circuits Magazine - Fall 2021 - 49
IEEE Solid-States Circuits Magazine - Fall 2021 - 50
IEEE Solid-States Circuits Magazine - Fall 2021 - 51
IEEE Solid-States Circuits Magazine - Fall 2021 - 52
IEEE Solid-States Circuits Magazine - Fall 2021 - 53
IEEE Solid-States Circuits Magazine - Fall 2021 - 54
IEEE Solid-States Circuits Magazine - Fall 2021 - 55
IEEE Solid-States Circuits Magazine - Fall 2021 - 56
IEEE Solid-States Circuits Magazine - Fall 2021 - 57
IEEE Solid-States Circuits Magazine - Fall 2021 - 58
IEEE Solid-States Circuits Magazine - Fall 2021 - 59
IEEE Solid-States Circuits Magazine - Fall 2021 - 60
IEEE Solid-States Circuits Magazine - Fall 2021 - 61
IEEE Solid-States Circuits Magazine - Fall 2021 - 62
IEEE Solid-States Circuits Magazine - Fall 2021 - 63
IEEE Solid-States Circuits Magazine - Fall 2021 - 64
IEEE Solid-States Circuits Magazine - Fall 2021 - 65
IEEE Solid-States Circuits Magazine - Fall 2021 - 66
IEEE Solid-States Circuits Magazine - Fall 2021 - 67
IEEE Solid-States Circuits Magazine - Fall 2021 - 68
IEEE Solid-States Circuits Magazine - Fall 2021 - 69
IEEE Solid-States Circuits Magazine - Fall 2021 - 70
IEEE Solid-States Circuits Magazine - Fall 2021 - 71
IEEE Solid-States Circuits Magazine - Fall 2021 - 72
IEEE Solid-States Circuits Magazine - Fall 2021 - 73
IEEE Solid-States Circuits Magazine - Fall 2021 - 74
IEEE Solid-States Circuits Magazine - Fall 2021 - 75
IEEE Solid-States Circuits Magazine - Fall 2021 - 76
IEEE Solid-States Circuits Magazine - Fall 2021 - 77
IEEE Solid-States Circuits Magazine - Fall 2021 - 78
IEEE Solid-States Circuits Magazine - Fall 2021 - 79
IEEE Solid-States Circuits Magazine - Fall 2021 - 80
IEEE Solid-States Circuits Magazine - Fall 2021 - 81
IEEE Solid-States Circuits Magazine - Fall 2021 - 82
IEEE Solid-States Circuits Magazine - Fall 2021 - 83
IEEE Solid-States Circuits Magazine - Fall 2021 - 84
IEEE Solid-States Circuits Magazine - Fall 2021 - 85
IEEE Solid-States Circuits Magazine - Fall 2021 - 86
IEEE Solid-States Circuits Magazine - Fall 2021 - 87
IEEE Solid-States Circuits Magazine - Fall 2021 - 88
IEEE Solid-States Circuits Magazine - Fall 2021 - 89
IEEE Solid-States Circuits Magazine - Fall 2021 - 90
IEEE Solid-States Circuits Magazine - Fall 2021 - 91
IEEE Solid-States Circuits Magazine - Fall 2021 - 92
IEEE Solid-States Circuits Magazine - Fall 2021 - 93
IEEE Solid-States Circuits Magazine - Fall 2021 - 94
IEEE Solid-States Circuits Magazine - Fall 2021 - 95
IEEE Solid-States Circuits Magazine - Fall 2021 - 96
IEEE Solid-States Circuits Magazine - Fall 2021 - 97
IEEE Solid-States Circuits Magazine - Fall 2021 - 98
IEEE Solid-States Circuits Magazine - Fall 2021 - 99
IEEE Solid-States Circuits Magazine - Fall 2021 - 100
IEEE Solid-States Circuits Magazine - Fall 2021 - 101
IEEE Solid-States Circuits Magazine - Fall 2021 - 102
IEEE Solid-States Circuits Magazine - Fall 2021 - 103
IEEE Solid-States Circuits Magazine - Fall 2021 - 104
IEEE Solid-States Circuits Magazine - Fall 2021 - 105
IEEE Solid-States Circuits Magazine - Fall 2021 - 106
IEEE Solid-States Circuits Magazine - Fall 2021 - 107
IEEE Solid-States Circuits Magazine - Fall 2021 - 108
IEEE Solid-States Circuits Magazine - Fall 2021 - 109
IEEE Solid-States Circuits Magazine - Fall 2021 - 110
IEEE Solid-States Circuits Magazine - Fall 2021 - 111
IEEE Solid-States Circuits Magazine - Fall 2021 - 112
IEEE Solid-States Circuits Magazine - Fall 2021 - 113
IEEE Solid-States Circuits Magazine - Fall 2021 - 114
IEEE Solid-States Circuits Magazine - Fall 2021 - 115
IEEE Solid-States Circuits Magazine - Fall 2021 - 116
IEEE Solid-States Circuits Magazine - Fall 2021 - 117
IEEE Solid-States Circuits Magazine - Fall 2021 - 118
IEEE Solid-States Circuits Magazine - Fall 2021 - 119
IEEE Solid-States Circuits Magazine - Fall 2021 - 120
IEEE Solid-States Circuits Magazine - Fall 2021 - 121
IEEE Solid-States Circuits Magazine - Fall 2021 - 122
IEEE Solid-States Circuits Magazine - Fall 2021 - 123
IEEE Solid-States Circuits Magazine - Fall 2021 - 124
IEEE Solid-States Circuits Magazine - Fall 2021 - 125
IEEE Solid-States Circuits Magazine - Fall 2021 - 126
IEEE Solid-States Circuits Magazine - Fall 2021 - 127
IEEE Solid-States Circuits Magazine - Fall 2021 - 128
IEEE Solid-States Circuits Magazine - Fall 2021 - 129
IEEE Solid-States Circuits Magazine - Fall 2021 - 130
IEEE Solid-States Circuits Magazine - Fall 2021 - 131
IEEE Solid-States Circuits Magazine - Fall 2021 - 132
IEEE Solid-States Circuits Magazine - Fall 2021 - 133
IEEE Solid-States Circuits Magazine - Fall 2021 - 134
IEEE Solid-States Circuits Magazine - Fall 2021 - 135
IEEE Solid-States Circuits Magazine - Fall 2021 - 136
IEEE Solid-States Circuits Magazine - Fall 2021 - 137
IEEE Solid-States Circuits Magazine - Fall 2021 - 138
IEEE Solid-States Circuits Magazine - Fall 2021 - 139
IEEE Solid-States Circuits Magazine - Fall 2021 - 140
IEEE Solid-States Circuits Magazine - Fall 2021 - 141
IEEE Solid-States Circuits Magazine - Fall 2021 - 142
IEEE Solid-States Circuits Magazine - Fall 2021 - 143
IEEE Solid-States Circuits Magazine - Fall 2021 - 144
IEEE Solid-States Circuits Magazine - Fall 2021 - 145
IEEE Solid-States Circuits Magazine - Fall 2021 - 146
IEEE Solid-States Circuits Magazine - Fall 2021 - 147
IEEE Solid-States Circuits Magazine - Fall 2021 - 148
IEEE Solid-States Circuits Magazine - Fall 2021 - 149
IEEE Solid-States Circuits Magazine - Fall 2021 - 150
IEEE Solid-States Circuits Magazine - Fall 2021 - 151
IEEE Solid-States Circuits Magazine - Fall 2021 - 152
IEEE Solid-States Circuits Magazine - Fall 2021 - 153
IEEE Solid-States Circuits Magazine - Fall 2021 - 154
IEEE Solid-States Circuits Magazine - Fall 2021 - 155
IEEE Solid-States Circuits Magazine - Fall 2021 - 156
IEEE Solid-States Circuits Magazine - Fall 2021 - 157
IEEE Solid-States Circuits Magazine - Fall 2021 - 158
IEEE Solid-States Circuits Magazine - Fall 2021 - 159
IEEE Solid-States Circuits Magazine - Fall 2021 - 160
IEEE Solid-States Circuits Magazine - Fall 2021 - 161
IEEE Solid-States Circuits Magazine - Fall 2021 - 162
IEEE Solid-States Circuits Magazine - Fall 2021 - 163
IEEE Solid-States Circuits Magazine - Fall 2021 - 164
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover3
IEEE Solid-States Circuits Magazine - Fall 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2023
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2022
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2021
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_spring2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_winter2020
https://www.nxtbook.com/nxtbooks/ieee/mssc_fall2019
https://www.nxtbook.com/nxtbooks/ieee/mssc_summer2019
https://www.nxtbook.com/nxtbooks/ieee/mssc_2019summer
https://www.nxtbook.com/nxtbooks/ieee/mssc_2019winter
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018fall
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018summer
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018spring
https://www.nxtbook.com/nxtbooks/ieee/mssc_2018winter
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2017
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2016
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2015
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_winter2014
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_fall2014
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_summer2014
https://www.nxtbook.com/nxtbooks/ieee/solidstatecircuits_spring2014
https://www.nxtbookmedia.com