Instrumentation & Measurement Magazine 23-8 - 21

Fig. 3. (a) A frame of a polarized image acquired at 18 Hz. (b) An unpolarized
image acquired at 18 Hz signal. Both were reconstructed with a 7500 μs time
window.

rotational frequency is highly dependent on time. The AEDAT
recordings used as input to the neural network were reconstructed using a 500 μs time window. Each window contains
the number of events that occurred during that time window.

Neural Network Description
Preliminary studies were based on full frame image reconstruction that made use of two neural networks in a single
pipeline to classify rotational frequency [10]. In those studies,
firstly, neuromorphic data was converted from AEDAT into
AVI video. Features were then extracted using a pre-trained
convolutional neural network (Inception V3) and were later
fed into a Long Short-Term Memory (LSTM) recurrent neural
network and a multilayer perceptron. That pipeline was very
complex and resource intensive to train, due in part to the complexity involved with extracting features from images and the
large size of the LSTM layers in the neural network.
In contrast, the neural network used in this study is relatively simple and contains three types of layers: average
pooling, GRU and dense. Average pooling reduces the data by
averaging time windows within one input together. This layer

was used at the beginning of the neural network to smooth out
the data and reduce the impact of outliers. GRU is a recurrent
layer that can perform temporal feature extraction. GRU aims
to solve the vanishing gradient problem of recurrent neural
networks. The dense layers with sigmoid activation functions
are used to further feature extraction but does not provide temporal based features. Overall, the neural network used in this
study is simple and does not take long to train or process data.
As stated above, training using time event windows as opposed to full frame images as input to a neural network is more
efficient and greatly reduces training time. A neural network
with time event windows as input takes approximately 1 minute to train, while a neural network with full frame images as
input may take 6 hours or longer to train.

Results and Discussion
A data processing pipeline was established to ensure each recording was handled in the same manner to ensure consistent
results. The pipeline processing steps are the following:
◗◗ Use JAER (Java Address-Event Representation) program
to capture 1.5-minute AEDAT recordings.
◗◗ Use a custom UWP (Universal Windows Platform)
AEDAT file reader to convert the files from a byte format
to a CSV format using time-based reconstruction.
◗◗ Use a Python script to group the 1.5-minute CSV files
into 500 ms segments. This is then fed into a deep neural
network.
Acquired polarized and unpolarized images are shown in
Fig. 3. The image in Fig. 3b shows blurring behind the blades of
the wheel that is not present in the polarized image. Polarization also improves the ability to distinguish between different
frequencies.
The polarization images exhibit better temporal contrast
with respect to unpolarized one. This is attributed to high scatter background rejection, leading to an enhanced sensitivity,
which improves the temporal resolution of the p(DVS) system.

Fig. 4. (a) Neural Network Validation Accuracy and (b) Validation Loss with sequence of 200 time event windows (500 μs /event window).
November 2020

IEEE Instrumentation & Measurement Magazine 21

Instrumentation & Measurement Magazine 23-8

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 23-8