The Bridge - Issue 2, 2018 - 13

Determining Optimum Drop-out Rate for Neural Networks

TensorFlow Accuracy Determination
and Decision Threshold
The model accuracy is an output of the "evaluate"
method within TensorFlow. It is calculated by first
feeding the testing set through the trained model.
The accuracy is then the number of correctly
predicted classifications divided by the number of
predictions made, or the size of the testing set.
The DNNLinearCombinedClassifier method within
TensorFlow is thought to use a default threshold
of 0.5 when training the DNN. This can be
changed by using the predict_proba method within
DNNLinearCombinedClassifier to return a predicted
probability for a given feature set, or sample. The
return predicted probability can then be compared
to a threshold and converted to the original binary
classification.
Datasets used in our experiments: The Credit
Card Default, Breast Cancer, and Bank Marketing
Datasets
Due to their prominence, datasets that are nontime series, non-spatial, and have heterogeneous
inputs are commonly used to create predictive
models. The Taiwanese credit card default dataset
(Yeh & Lien 2009) used in this study contains client
attributes such as education level, marriage status,
and past payments for approximately 30,000 clients.
In addition, it includes whether or not each client
defaulted on the credit card, which is necessary to
train the model.

The bank marketing dataset (Moro, Cortez, & Rita
2014) includes each clients financial background,
marital status, amount of exposure of marketing
campaigns, etc. and whether or not they subscribed
to a term deposit. Models trained on this dataset
contained a training size of 28,000 samples and a
testing size of 15,000 samples.
Visually evaluating the separability of the credit
card default dataset
A separable dataset is one which can be classified
perfectly - a surface exists in the feature space
which divides the categories perfectly.
To evaluate the separability of the credit card default
dataset visually, a neural network was trained on
just three of the input variables: age, pay0, and
pay2 (The dataset does not include a pay1 field).
The attributes pay0 and pay2 each represent the
payment status of a month before the possible
default. Negative values represent payment that was
on time, and positive values represent the number
of months a payment was late. Figure 3 shows the
trained network's decision surfaces with various
thresholds. The credit card data is far from separable
when only these inputs are considered. There are
many points with exactly the same inputs but both
default and non-default outputs. Pay0 is a much
stronger indicator of default than pay3 or age; the
vertical decision lines show that changing the second
variable does not change the prediction significantly.

The breast cancer dataset (Wolberg & Mangasarian
1990) includes cancerous cell size uniformity, other
diagnosis metrics, as well as the definitive diagnosis
of whether a cancerous clump exists. This dataset
consists of 699 samples.

HKN.ORG

http://www.HKN.ORG

Table of Contents for the Digital Edition of The Bridge - Issue 2, 2018