IEEE Consumer Electronics Magazine - March 2018 - 56

The engineer, approaching this
problem from a broader perspective, would like to be able to
combine multiple DNN techniques,
leveraging the positive benefits
of each.
Rarely does a new technique or tactic perform so well that all
previous methods become obsolete. However, the engineer,
approaching this problem from a broader perspective, would
like to be able to combine multiple DNN techniques, leveraging
the positive benefits of each and mitigating their weaker aspects.
A second challenge is that academic research is rarely
concerned with optimizing a technique for size or architectural efficiency. Typically, many resources are directed to the
narrow problem at hand, and even where optimizations are
investigated-and, thankfully, the deep-learning research
community has become more sensitive to their importance-
they apply to only a particular DNN architecture. In general,
researchers are focused on designing a network to solve an
immediate problem and do not think more widely about sharing structures or resources between multiple DNNs.
Thus, for the engineer approaching a particular problem in
machine learning and seeking to adopt techniques from the literature, there will typically be a handful of different DNN
solutions available, each adopting different network structures
and optimizations while still sharing many common network
layers and data inputs. However, there has been little research
concerned with techniques for merging and optimizing a combination of existing DNN approaches into a more holistic solution. In effect, almost everyone seems to be focused on
growing an insular cluster of trees and developing these as
quickly as possible, rather than working with others to develop
and manage a forest where everyone can take advantage of
economies of scale.
Fortunately, most DNN architectures do share many common elements, just as many parts of a forest share similar
trees. Furthermore, it is possible to bring such elements together and develop more accurate, optimized, and improved
DNNs that will deliver better performance and economies of
scale with efficient implementations that are well suited for
CE deployments. This is what we will focus on in the remainder of this article.

A CRASH COURSE IN DEEP NETWORKS
Here, we provide a quick refresher on some of the most important elements of a DNN. For those who are unfamiliar with
deep learning, our previous article [2] is recommended. DNN
networks comprise several signal-processing methods applied
in sequence to an input data structure. In image-understanding tasks, these signal-processing elements are typically convolutional and fully connected layers, followed by pooling
56 IEEE Consumer Electronics Magazine

MARCH 2018

and regularization layers. Data flow through these layers in
sequence, eventually emerging in a modified structure that
typically enables a binary interpretation of some feature or
characteristic of the input data structure. Typically, the chain
of data processing layers is much longer (deeper) than in classical neural networks, hence the descriptive term deep that is
applied to such networks.

THE MAIN CATEGORIES OF DEEP NETWORK LAYERS
Deep networks comprise a succession of data-processing layers, each taking inputs from a preceding layer and filtering a
set of inputs to generate a set of outputs.
▼▼ Convolutional layers: These are the best-known layers;
they convolve the input data from the previous layer I with
a kernel W. In the general case, an offset bias b is added to
this convolution as follows:
P = I ) W + b.
In computer-vision applications, the original input I is typically a structured set of image pixels, but as the original data
element is processed by successive layers of the network, it
becomes altered and transformed until, eventually, a much
simpler output is obtained from the network output. Often,
this is a simple binary decision, although other forms of output can also be generated.
Other layer types are also used in deep network architectures, such as the following.
▼▼ Pooling layers: These typically apply a nonlinear transform on the input data that is normally used to reduce the
size of the input data. Such layers can be thought of as
data concentrator elements, and they are important for controlling the overall size of a network, which could grow
too large if only convolutional layers were used.
▼▼ Fully connected layers: These are exactly the same as classical neural network layers, where all of the neurons in a
layer are connected to all of the neurons in the subsequent
layer. The neurons give the summation of their input multiplied by their weights, which is then passed through their
activation functions. Even more than with convolutional
layers, these can cause the network size to grow, and so,
typically, only one or two fully connected layers will be
used in most deep networks.
▼▼ Regularization layers: These are used to prevent overfitting inside the network. Different kinds of regularizations
have been proposed, the most important ones being 1)
weight regularization, 2) dropout regularization, where
some data are skipped, and 3) batch normalization, where
output data are averaged across several input data. Each
of these regularization techniques has advantages and
drawbacks that make them more (or less) suitable for specific applications.

THE COSTS OF GOING DEEP
As was previously mentioned, there has been spectacular
growth in research on AI in general and on the application

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - March 2018