IEEE Computational Intelligence Magazine - November 2020 - 44

A. Long Short Term Memory (LSTM) Cells

The output value of the LSTM unit at time t is

An LSTM unit consists of three gates (i.e., update, forget, and
output gates) and three cells (i.e., input, memory, and update
cells). The memory cell at time t is updated using a candidate
value uc 1 t 2, which is calculated using the output value at time
t - 1, i.e., a 1 t - 1 2, and input value at time t, i.e., x 1 t 2, through
the equation
	

uc 1 t 2 = tanh ^W c 6a 1 t - 1 2, x 1 t 2@ + b c h (6)

where tanh (.) is the hyperbolic tangent activation function,
and W c and b c represent the matrix of parameters and biased
vector of the memory cell, respectively. The value of the memory cell c 1 t 2 is then updated using the candidate value uc 1 t 2
and the previous value c 1 t - 1 2 through
	

c 1 t 2 = C u 9 uc 1 t 2 + C f 9 c 1 t - 1 2 (7)

where 9 indicates element-wise multiplication. C u and C f are
the values of the update and forget gates which are obtained from
	

C u = v ^W u 6a 1 t - 1 2, x 1 t 2@ + b uh (8)

and
	

C f = v ^W f 6a 1 t - 1 2, x 1 t 2@ + b f h (9)

in which v (.) is the sigmoid activation function, W u and b u
are respectively the matrix of parameters and the bias vector
corresponding to the update gate, and W f and b f are respectively the matrix of parameters and the bias vector corresponding to the forget gate.

a 1 t 2 = C o 9 tanh ^c 1 t 2h (10)

	

where C o is the value of the output gate which itself is
C q = v ^W o 6a 1 t - 1 2, x 1 t 2@ + b oh (11)

	

in which W o and b o are respectively the matrix of parameters
and the bias vector corresponding to the output gate. Figure 12 shows an LSTM unit.
A multivariate RNN architecture is used in this paper,
which takes multiple features as input, and outputs the predicted value. Two different architectures are used, one for training
CC-IMF1, and another for training CC-IMF2 and CC-IMF3
separately. The architecture of the stacked RNN corresponding
to the forecasting future value of CC-IMF1 is as follows:
1)	a sequence input layer which accepts the number of
inputs equal to the number of features. Regarding Maharashtra, there are two features for training CC-IMF1: the
signals CC-IMF1 and TR at time t - 1. For Tamil Nadu
there are three features for training CC-IMF1: the signals
CC-IMF1, H-IMF1 and TR at time t - 1. The value of
the CC-IMF1 at time t is the target value which needs to
be predicted in both cases.
2)	an LSTM layer with 50 units.
3)	a dropout layer with the factor 0.6.
4)	a fully connected layer with one output unit.
The architecture used for training CC-IMF2 and CC-IMF3
is as follows:

y 

softmax

c

c

c

tanh

∼
c

a

f
Forget Gate





u
Update Gate

a

o
tanh

Output Gate

x 

FIGURE 12 Visualisation of an LSTM unit. y 1 t 2 is the final output of an LSTM unit at time t which is computed by a softmax activation function.

44

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2020
IEEE Computational Intelligence Magazine - November 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2020
