Content Gazette - June 2020 - 30

Contents

Volume 37 | Number 3 | May 2020

SPECIAL SECTION

102

DISTRIBUTED, STREAMING
MACHINE LEARNING
11
14

Ran Xin, Soummya Kar,
and Usman A. Khan

114

FROM THE GUEST EDITORS

Waheed U. Bajwa, Volkan Cevher,
Dimitris Papailiopoulos,
and Anna Scaglione

123

MULTITASK LEARNING
OVER GRAPHS

DISTRIBUTED LEARNING IN THE
NONCONVEX WORLD

Tsung-Hui Chang, Mingyi Hong,
Hoi-To Wai, Xinwei Zhang,
and Songtao Lu

39

50

DISTRIBUTED TRAINING OF DEEP NEURAL
NETWORK ACOUSTIC MODELS FOR
AUTOMATIC SPEECH RECOGNITION

ON THE COVER
This special issue on distributed, streaming machine
learning presents recent advances in several topic areas
that pertain to the training of machine learning models
from data that are distributed, streaming, or both distributed and streaming. A particular emphasis of articles in
the special issue is to provide the reader with an entry
point into algorithmic and analytical techniques that may
be relevant to the industry for the emerging era of realtime, decentralized, and autonomous decision making.

Xiaodong Cui, Wei Zhang,
Ulrich Finkler, George Saon,
Michael Picheny, and David Kung

COVER IMAGE: ©ISTOCKPHOTO.COM/MF3D

FEDERATED LEARNING

61

Tian Li, Anit Kumar Sahu,
Ameet Talwalkar, and Virginia Smith

OPTIMALLY COMPRESSED
NONPARAMETRIC ONLINE LEARNING

OPTIMIZATION FOR
REINFORCEMENT LEARNING
Donghwan Lee, Niao He,
Parameswaran Kamalaruban,
and Volkan Cevher

136

STRAGGLER-RESISTANT
DISTRIBUTED MATRIX
COMPUTATION VIA CODING THEORY
Aditya Ramamoorthy,
Anindya Bijoy Das, and Li Tang

146

ADVERSARY-RESILIENT
DISTRIBUTED AND DECENTRALIZED
STATISTICAL INFERENCE AND
MACHINE LEARNING
Zhixiong Yang, Arpita Gang,
and Waheed U. Bajwa

Alec Koppel, Amrit Singh Bedi,
Ketan Rajawat, and Brian M. Sadler

71

OPTIMIZATION AND LEARNING WITH
INFORMATION STREAMS
Emiliano Dall'Anese,
Andrea Simonetto, Stephen Becker,
and Liam Madden

84

DISTRIBUTED NO-REGRET LEARNING
IN MULTIAGENT SYSTEMS
Xiao Xu and Qing Zhao

92
PG. 8

ASYMPTOTIC NETWORK
INDEPENDENCE IN DISTRIBUTED
STOCHASTIC OPTIMIZATION
FOR MACHINE LEARNING
Shi Pu, Alex Olshevsky,
and Ioannis Ch. Paschalidis

Roula Nassif, Stefan Vlaski,
Cédric Richard, Jie Chen,
and Ali H. Sayed

26

DECENTRALIZED STOCHASTIC
OPTIMIZATION AND MACHINE
LEARNING

DISTRIBUTED GRADIENT METHODS
FOR CONVEX MACHINE LEARNING
PROBLEMS IN NETWORKS
Angelia Nedic´

Reinvestment
to Innovation

Profit
Generation
From the
Innovation

Industry
Request for
Innovation

Development
of New
Standard

Releasing
Standard to
Market
Deployment of
the Standard

PG. 160

IEEE SIGNAL PROCESSING MAGAZINE (ISSN 1053-5888) (ISPREG) is published bimonthly by the Institute of Electrical and Electronics Engineers, Inc., 3 Park Avenue, 17th Floor, New York,
NY 10016-5997 USA (+1 212 419 7900). Responsibility for the contents rests upon the authors and not the IEEE, the Society, or its members. Annual member subscriptions included in Society fee.
Nonmember subscriptions available upon request. Individual copies: IEEE Members US$20.00 (first copy only), nonmembers US$246 per copy. Copyright and Reprint Permissions: Abstracting
is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright Law for private use of patrons: 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without fee. Instructors are
permitted to photocopy isolated articles for noncommercial classroom use without fee. For all other copying, reprint, or republication permission, write to IEEE Service Center, 445 Hoes Lane,
Piscataway, NJ 08854 USA. Copyright © 2020 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals postage paid at New York, NY, and at additional mailing offices.
Postmaster: Send address changes to IEEE Signal Processing Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188
Printed in the U.S.A.

Digital Object Identifier 10.1109/MSP.2020.2971869

www.signalprocessingsociety.org
JUNE
IEEESIGNAL
SIGNALPROCESSING
PROCESSINGMAGAZINE
MAGAZINE ||[30
May] 2020
|| 2020
IEEE
March
2020

1


https://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=79 https://ieeexplore.ieee.org/document/9084360 https://ieeexplore.ieee.org/document/9084351 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9084337 https://ieeexplore.ieee.org/document/9084370 https://ieeexplore.ieee.org/document/9084325 https://ieeexplore.ieee.org/document/9085431 https://ieeexplore.ieee.org/document/9084368 https://ieeexplore.ieee.org/document/9084348 https://ieeexplore.ieee.org/document/9084329 http://www.ISTOCKPHOTO.COM/MF3D https://ieeexplore.ieee.org/document/9084339 https://ieeexplore.ieee.org/document/9084352 https://ieeexplore.ieee.org/document/9085968 https://ieeexplore.ieee.org/document/9084341 https://ieeexplore.ieee.org/document/9084356 https://ieeexplore.ieee.org/document/9084373 https://ieeexplore.ieee.org/document/9084372 http://www.signalprocessingsociety.org

Content Gazette - June 2020

Table of Contents for the Digital Edition of Content Gazette - June 2020

Content Gazette - June 2020 - Cover1
Content Gazette - June 2020 - Cover2
Content Gazette - June 2020 - 1
Content Gazette - June 2020 - 2
Content Gazette - June 2020 - 3
Content Gazette - June 2020 - 4
Content Gazette - June 2020 - 5
Content Gazette - June 2020 - 6
Content Gazette - June 2020 - 7
Content Gazette - June 2020 - 8
Content Gazette - June 2020 - 9
Content Gazette - June 2020 - 10
Content Gazette - June 2020 - 11
Content Gazette - June 2020 - 12
Content Gazette - June 2020 - 13
Content Gazette - June 2020 - 14
Content Gazette - June 2020 - 15
Content Gazette - June 2020 - 16
Content Gazette - June 2020 - 17
Content Gazette - June 2020 - 18
Content Gazette - June 2020 - 19
Content Gazette - June 2020 - 20
Content Gazette - June 2020 - 21
Content Gazette - June 2020 - 22
Content Gazette - June 2020 - 23
Content Gazette - June 2020 - 24
Content Gazette - June 2020 - 25
Content Gazette - June 2020 - 26
Content Gazette - June 2020 - 27
Content Gazette - June 2020 - 28
Content Gazette - June 2020 - 29
Content Gazette - June 2020 - 30
Content Gazette - June 2020 - 31
Content Gazette - June 2020 - 32
Content Gazette - June 2020 - 33
Content Gazette - June 2020 - 34
Content Gazette - June 2020 - 35
Content Gazette - June 2020 - 36
Content Gazette - June 2020 - 37
Content Gazette - June 2020 - 38
Content Gazette - June 2020 - Cover3
Content Gazette - June 2020 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202105
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202104
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202103
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202102
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202101
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202012
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202011
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202010
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202009
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202008
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202007
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202006
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202005
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202004
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202003
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202002
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_202001
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201912
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201911
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201910
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201909
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201908
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201907
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201906
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201905
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201904
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201903
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201902
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201901
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201812
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201811
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201810
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201809
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201808
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201807
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201806
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201805
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201804
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201803
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201802
https://www.nxtbook.com/nxtbooks/ieee/contentgazette_201801
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1217
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1117
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1017
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0917
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0817
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0717
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0617
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0517
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0417
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0317
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0217
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0117
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1216
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1116
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1016
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0916
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0816
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0716
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0616
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0516
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0416
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0316
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0216
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0116
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1215
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1115
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1015
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0915
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0815
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0715
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0615
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0515
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0415
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0315
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0215
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0115
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1214
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1114
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1014
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0914
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0814
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0714
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0614
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0514
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0414
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0314
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0214
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0114
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1213
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1113
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1013
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0913
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0813
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0713
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0613
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0513
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0413
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0313
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0213
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0113
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1212
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1112
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1012
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0912
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0812
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0712
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0612
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0512
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0412
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0312
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0212
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0112
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1211
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1111
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_1011
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0911
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0811
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0711
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0611
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0511
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0411
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0311
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0211
https://www.nxtbook.com/nxtbooks/ieee/content_gazette_0111
https://www.nxtbookmedia.com