Instrumentation & Measurement Magazine 24-6 - 24

Trainee Environment
The trainees use the coach's video and the camera of their EC
device to track their pose. The camera feed is passed to a light
pre-trained pose estimation model, a MobileNet. We use the
keypoints generated from this light model and the coach's
keypoints to train an SNN to learn from the trainee's intention.
We can then use several post-processing calculations to
fix any troublesome keypoints. Finally, we use the fixed trainee's
keypoints and the coach's keypoints to give real-time
feedback to the trainee using their Angular Pose Representation
(APR).
Neural Network for Pose Estimation
Context Tuning
Pose Estimation Pre-Trained Model
For pose estimation, we use the PoseNet model based on MobileNet
for the trainee and ResNet50 for the coach. The internal
PoseNet model is based on the work in [17], a CNN model
trained on the COCO database.
Data Collection
There are currently no datasets for coach-trainee videos.
Therefore, we decided to do our data collection for the measurements.
The research experiments have been approved by
the University of Ottawa Research Ethics Board, File Number:
[H-08-20-5859].
We asked 10 participants (seven males and three females,
aged 22 to 58 years) to follow a pre-recorded 5-minute routine
video composed of stretching and squatting exercises
over one month using our proof-of-concept application DTCoach.
With this app, we collected the pose estimation output
keypoints and saved them into a cloud database. At the end
of the one month, we ended up with 38 different routines collected,
which are more than 110,000 single frames for training
since there are about 2,900 frames per routine. We randomly
selected five videos for validation and 33 videos for training.
From these 33 videos, we made a 0.67-0.33 train-test split for
the training process.
Pre-Processing
Before we can use DL algorithms on the measured data, we
need to calibrate our data to make it easier to process. We will
only use 14 keypoints, shown in Fig. 2, since we are only interested
in key joints.
There are five steps involved in the calibration process:
◗ Centralize to origin (0,0): First, we centralized the
keypoints into the origin since the person will not always
be at the center of the frame.
◗ Mirroring: We fix the orientation of the pose caused by
the way pixels are counted in images and videos from the
frame's upper left side.
◗ Normalization (-1,1): Now, we normalize into a range of
-1 to 1.
◗ Flatten into 1D array: This way, we can feed it into the
SNN.
24
Fig. 2. Fourteen keypoints that will be used for the SNN.
◗ Synchronization: When the trainees perform the pose,
they see in the routine video, and there will be a slight
delay for them to follow what they see. We need to
compensate for this lag before comparing the trainee's
frame with the coach's frame. Therefore, in a window of
41 frames, we search for the most similar one to the trainee
pose in the coach video using the R2
score as the measurement
of similarity. Fig. 3 shows the difference in Mean
Squared Error (MSE) while training the 2-Layer SNN
configuration for 100 epochs. We can see that training
with sync reduces the MSE in the test and train datasets.
Therefore, we will be using synchronization since it
achieved better results.
Random Search Optimization
We used 42 features to train the SNN: 28, which are the coordinates
x and y of the 14 keypoints of the athlete, and the 14
confidence scores of these keypoints. As the output of the SNN
we used the 28 keypoints coordinates of the coach. We used
the Adam optimizer with a learning rate of 0.003 and MSE as
the loss function.
The SNN architecture was selected through Random Search
Optimization (RSO), a method where we run several test cases
with randomized hyperparameters and choose the configuration
that achieved the best results. We performed RSO with
six different configurations of hidden layers: 1, 2, 3, 4, 5, and 10
hidden layers. Each layer had the possibility of having from 1
to 100 nodes. Table 1 shows the results. From the 50 trials, we
show the average MSE and time per sample of the top 10 trials.
We also show the best trial configuration per layer count and
improvement in parameter count compared to the layer above.
We chose the 2-Layer (2-L) configuration because it had
better performance for the number of trainable parameters to
have a light model. We can also see that it was the one that took
IEEE Instrumentation & Measurement Magazine
September 2021

Instrumentation & Measurement Magazine 24-6

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-6

No label
Instrumentation & Measurement Magazine 24-6 - No label
Instrumentation & Measurement Magazine 24-6 - Cover2
Instrumentation & Measurement Magazine 24-6 - 1
Instrumentation & Measurement Magazine 24-6 - 2
Instrumentation & Measurement Magazine 24-6 - 3
Instrumentation & Measurement Magazine 24-6 - 4
Instrumentation & Measurement Magazine 24-6 - 5
Instrumentation & Measurement Magazine 24-6 - 6
Instrumentation & Measurement Magazine 24-6 - 7
Instrumentation & Measurement Magazine 24-6 - 8
Instrumentation & Measurement Magazine 24-6 - 9
Instrumentation & Measurement Magazine 24-6 - 10
Instrumentation & Measurement Magazine 24-6 - 11
Instrumentation & Measurement Magazine 24-6 - 12
Instrumentation & Measurement Magazine 24-6 - 13
Instrumentation & Measurement Magazine 24-6 - 14
Instrumentation & Measurement Magazine 24-6 - 15
Instrumentation & Measurement Magazine 24-6 - 16
Instrumentation & Measurement Magazine 24-6 - 17
Instrumentation & Measurement Magazine 24-6 - 18
Instrumentation & Measurement Magazine 24-6 - 19
Instrumentation & Measurement Magazine 24-6 - 20
Instrumentation & Measurement Magazine 24-6 - 21
Instrumentation & Measurement Magazine 24-6 - 22
Instrumentation & Measurement Magazine 24-6 - 23
Instrumentation & Measurement Magazine 24-6 - 24
Instrumentation & Measurement Magazine 24-6 - 25
Instrumentation & Measurement Magazine 24-6 - 26
Instrumentation & Measurement Magazine 24-6 - 27
Instrumentation & Measurement Magazine 24-6 - 28
Instrumentation & Measurement Magazine 24-6 - 29
Instrumentation & Measurement Magazine 24-6 - 30
Instrumentation & Measurement Magazine 24-6 - 31
Instrumentation & Measurement Magazine 24-6 - 32
Instrumentation & Measurement Magazine 24-6 - 33
Instrumentation & Measurement Magazine 24-6 - 34
Instrumentation & Measurement Magazine 24-6 - 35
Instrumentation & Measurement Magazine 24-6 - 36
Instrumentation & Measurement Magazine 24-6 - 37
Instrumentation & Measurement Magazine 24-6 - 38
Instrumentation & Measurement Magazine 24-6 - 39
Instrumentation & Measurement Magazine 24-6 - 40
Instrumentation & Measurement Magazine 24-6 - 41
Instrumentation & Measurement Magazine 24-6 - 42
Instrumentation & Measurement Magazine 24-6 - 43
Instrumentation & Measurement Magazine 24-6 - 44
Instrumentation & Measurement Magazine 24-6 - 45
Instrumentation & Measurement Magazine 24-6 - 46
Instrumentation & Measurement Magazine 24-6 - 47
Instrumentation & Measurement Magazine 24-6 - 48
Instrumentation & Measurement Magazine 24-6 - 49
Instrumentation & Measurement Magazine 24-6 - 50
Instrumentation & Measurement Magazine 24-6 - 51
Instrumentation & Measurement Magazine 24-6 - 52
Instrumentation & Measurement Magazine 24-6 - 53
Instrumentation & Measurement Magazine 24-6 - 54
Instrumentation & Measurement Magazine 24-6 - 55
Instrumentation & Measurement Magazine 24-6 - 56
Instrumentation & Measurement Magazine 24-6 - 57
Instrumentation & Measurement Magazine 24-6 - 58
Instrumentation & Measurement Magazine 24-6 - 59
Instrumentation & Measurement Magazine 24-6 - 60
Instrumentation & Measurement Magazine 24-6 - 61
Instrumentation & Measurement Magazine 24-6 - 62
Instrumentation & Measurement Magazine 24-6 - 63
Instrumentation & Measurement Magazine 24-6 - 64
Instrumentation & Measurement Magazine 24-6 - 65
Instrumentation & Measurement Magazine 24-6 - 66
Instrumentation & Measurement Magazine 24-6 - 67
Instrumentation & Measurement Magazine 24-6 - 68
Instrumentation & Measurement Magazine 24-6 - 69
Instrumentation & Measurement Magazine 24-6 - 70
Instrumentation & Measurement Magazine 24-6 - 71
Instrumentation & Measurement Magazine 24-6 - 72
Instrumentation & Measurement Magazine 24-6 - 73
Instrumentation & Measurement Magazine 24-6 - 74
Instrumentation & Measurement Magazine 24-6 - 75
Instrumentation & Measurement Magazine 24-6 - 76
Instrumentation & Measurement Magazine 24-6 - 77
Instrumentation & Measurement Magazine 24-6 - 78
Instrumentation & Measurement Magazine 24-6 - 79
Instrumentation & Measurement Magazine 24-6 - 80
Instrumentation & Measurement Magazine 24-6 - 81
Instrumentation & Measurement Magazine 24-6 - 82
Instrumentation & Measurement Magazine 24-6 - 83
Instrumentation & Measurement Magazine 24-6 - 84
Instrumentation & Measurement Magazine 24-6 - 85
Instrumentation & Measurement Magazine 24-6 - 86
Instrumentation & Measurement Magazine 24-6 - 87
Instrumentation & Measurement Magazine 24-6 - 88
Instrumentation & Measurement Magazine 24-6 - 89
Instrumentation & Measurement Magazine 24-6 - 90
Instrumentation & Measurement Magazine 24-6 - 91
Instrumentation & Measurement Magazine 24-6 - 92
Instrumentation & Measurement Magazine 24-6 - 93
Instrumentation & Measurement Magazine 24-6 - 94
Instrumentation & Measurement Magazine 24-6 - 95
Instrumentation & Measurement Magazine 24-6 - 96
Instrumentation & Measurement Magazine 24-6 - Cover3
Instrumentation & Measurement Magazine 24-6 - Cover4
https://www.nxtbook.com/allen/iamm/26-6
https://www.nxtbook.com/allen/iamm/26-5
https://www.nxtbook.com/allen/iamm/26-4
https://www.nxtbook.com/allen/iamm/26-3
https://www.nxtbook.com/allen/iamm/26-2
https://www.nxtbook.com/allen/iamm/26-1
https://www.nxtbook.com/allen/iamm/25-9
https://www.nxtbook.com/allen/iamm/25-8
https://www.nxtbook.com/allen/iamm/25-7
https://www.nxtbook.com/allen/iamm/25-6
https://www.nxtbook.com/allen/iamm/25-5
https://www.nxtbook.com/allen/iamm/25-4
https://www.nxtbook.com/allen/iamm/25-3
https://www.nxtbook.com/allen/iamm/instrumentation-measurement-magazine-25-2
https://www.nxtbook.com/allen/iamm/25-1
https://www.nxtbook.com/allen/iamm/24-9
https://www.nxtbook.com/allen/iamm/24-7
https://www.nxtbook.com/allen/iamm/24-8
https://www.nxtbook.com/allen/iamm/24-6
https://www.nxtbook.com/allen/iamm/24-5
https://www.nxtbook.com/allen/iamm/24-4
https://www.nxtbook.com/allen/iamm/24-3
https://www.nxtbook.com/allen/iamm/24-2
https://www.nxtbook.com/allen/iamm/24-1
https://www.nxtbook.com/allen/iamm/23-9
https://www.nxtbook.com/allen/iamm/23-8
https://www.nxtbook.com/allen/iamm/23-6
https://www.nxtbook.com/allen/iamm/23-5
https://www.nxtbook.com/allen/iamm/23-2
https://www.nxtbook.com/allen/iamm/23-3
https://www.nxtbook.com/allen/iamm/23-4
https://www.nxtbookmedia.com