The Bridge - Issue 2, 2018 - 10

Feature

Determining Optimum
Drop-out Rate for
Neural Networks
by: Josiah A. Yoder
Department of Electrical Engineering
and Computer Science
Milwaukee School of Engineering
1025 North Broadway
Milwaukee WI 53202-3109
yoder@msoe.edu
Undergraduate Research Project

ABSTRACT
Dropout is used to reduce overfitting in neural
networks. Past research determines the optimum
dropout rate for a dataset but does not compare
optimal dropout rates across datasets. The purpose
of this project is to investigate a correlation
in optimum dropout rates between datasets
that are non-spatial, non-time series, and have
heterogeneous inputs. One dataset with these
properties is credit card default data, which contains
each client's age, education, etc., and whether they
defaulted on their credit card. A dropout rate of
0.5 is widely used but does not always optimize
performance. For each dataset, deep neural network
models were trained over various dropout rates
and training-set sizes. The experimental results
presented here show that the optimum dropout rate
falls anywhere within its possible range from 0 to
1, that even 10% dropout can significantly improve
performance over no dropout, and that dropout can
be effective even on small datasets.

within many of today's datasets, scientists often
apply neural networks and other machine learning
algorithms to detect complex nonlinear relationships
in a dataset. One drawback to expressive networks
is overfitting. Deep neural networks are very
expressive, but that expressiveness allows them to
overfit the training data, learning subtle distinctions
that are merely artifacts of some particular training
set. Figure 1a shows an example of overfitting. The
samples deviate from the true function, or signal,
because of noise, an effect that is common in real
world applications. The overfitting model shown by
the contour lines fits so well to the data that it also
captures the noise. Such a model would perform
extremely well during training but would perform
poorly during testing or live implementation. A more
versatile and useful model is shown in Figure 1b.

(a)

(b)

INTRODUCTION
Data is more available than ever before. However,
basic data analysis often does not result in accurate
predictive models. To model the complex patterns

THE BRIDGE

Figure 1: An example of an overfitted model. (a) A neural
network trained with 100 nodes on the PAY0 and PAY2
fields of the credit card default data set. (b) A neural



Table of Contents for the Digital Edition of The Bridge - Issue 2, 2018

Contents
The Bridge - Issue 2, 2018 - Cover1
The Bridge - Issue 2, 2018 - Cover2
The Bridge - Issue 2, 2018 - Contents
The Bridge - Issue 2, 2018 - 4
The Bridge - Issue 2, 2018 - 5
The Bridge - Issue 2, 2018 - 6
The Bridge - Issue 2, 2018 - 7
The Bridge - Issue 2, 2018 - 8
The Bridge - Issue 2, 2018 - 9
The Bridge - Issue 2, 2018 - 10
The Bridge - Issue 2, 2018 - 11
The Bridge - Issue 2, 2018 - 12
The Bridge - Issue 2, 2018 - 13
The Bridge - Issue 2, 2018 - 14
The Bridge - Issue 2, 2018 - 15
The Bridge - Issue 2, 2018 - 16
The Bridge - Issue 2, 2018 - 17
The Bridge - Issue 2, 2018 - 18
The Bridge - Issue 2, 2018 - 19
The Bridge - Issue 2, 2018 - 20
The Bridge - Issue 2, 2018 - 21
The Bridge - Issue 2, 2018 - 22
The Bridge - Issue 2, 2018 - 23
The Bridge - Issue 2, 2018 - 24
The Bridge - Issue 2, 2018 - 25
The Bridge - Issue 2, 2018 - 26
The Bridge - Issue 2, 2018 - 27
The Bridge - Issue 2, 2018 - 28
The Bridge - Issue 2, 2018 - 29
The Bridge - Issue 2, 2018 - 30
The Bridge - Issue 2, 2018 - 31
The Bridge - Issue 2, 2018 - 32
The Bridge - Issue 2, 2018 - 33
The Bridge - Issue 2, 2018 - 34
The Bridge - Issue 2, 2018 - 35
The Bridge - Issue 2, 2018 - 36
The Bridge - Issue 2, 2018 - 37
The Bridge - Issue 2, 2018 - 38
The Bridge - Issue 2, 2018 - 39
The Bridge - Issue 2, 2018 - 40
The Bridge - Issue 2, 2018 - 41
The Bridge - Issue 2, 2018 - 42
The Bridge - Issue 2, 2018 - 43
The Bridge - Issue 2, 2018 - 44
https://www.nxtbook.com/nxtbooks/ieee/bridge_issue2_2022
https://www.nxtbook.com/nxtbooks/ieee/bridge_issue1_2022
https://www.nxtbook.com/nxtbooks/ieee/bridge_issue3_2021
https://www.nxtbook.com/nxtbooks/ieee/bridge_issue2_2021
https://www.nxtbook.com/nxtbooks/ieee/bridge_issue1_2021
https://www.nxtbook.com/nxtbooks/ieee/bridge_2020_issue3
https://www.nxtbook.com/nxtbooks/ieee/bridge_2020_issue2
https://www.nxtbook.com/nxtbooks/ieee/bridge_2020_issue1
https://www.nxtbook.com/nxtbooks/ieee/bridge_2019_issue3
https://www.nxtbook.com/nxtbooks/ieee/bridge_2019_issue2
https://www.nxtbook.com/nxtbooks/ieee/bridge_2019_issue1
https://www.nxtbook.com/nxtbooks/ieee/bridge_2018_issue3
https://www.nxtbook.com/nxtbooks/ieee/bridge_2018_issue2
https://www.nxtbook.com/nxtbooks/ieee/bridge_2018_issue1
https://www.nxtbookmedia.com